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ABSTRACT 

The Joint federal/State Task Force on Evaluation 
(the Belmont Group) is currently developing a Joint Comprehensive 
Evaluation System (JCES) for the federally funded programs under the 
federal acts listed above. It is planned that JCES will replace the 
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project and emphasizes a systems analytic viewpoint. One of the 
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the substantive programs being funded. The JCES evaluation model is 
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system reporting requirements are discussed and specified. The data 
analysis plan requirements are outlined. JCES implementation 
schedules, working organization, and manpower requirements Are noted. 
Political and practical implementation problems are discussed. The 
sampling designs to be jed are described in detail. Finally, brief 
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Thin report covers Phase II in a continuing effort to develop 
a comprehensive review and critical examination of the Comprehensive 
Evaluation System being developed by The Joint Federal/State Task 
Force on Evaluation. The first (Phase I) report was primarily 
descriptive of the various aspects and purposes of the JCES as of 
that date (22 November 1969). It attempted to review the JCES in 
some detail, but provided only the beginnings of integrative and 
critical comment. 

The present document presents a more refined view of the JCES, 
emphasising a systems analytic viewpoint. Since the system is con- 
stantly evolving, It has been necessary to bring the review up to 
date to describe the system as it is envisioned as of the present 
time. The current system description ani review Incorporates many 
of the analytic data and conclusions derived from ongoing analytic 
studies. Ihis is especially true in the areas of evaluation 
concepts, systems concepts And data analysis and reporting require- 
ments. 

Inclusion of these data has yielded a much more comprehensive 
view of the JCES and its future developmental requirements than was 
possible during Phase I of the study. 

In future phases of this work, it is expected that the systems 
concepts, and the reporting and data analysis plans will be fully 
developed. 

SES wishes to express its appreciation for the help of many of 
the USOE Staff in the collection of data pertinent to the prepsratlon 
of this document. 



INTRODUCTION 

The lsst two decades have witnessed increasing concern 
with the effectiveness and efficiency of the American education* 
al process. This growing concern has led naturally to the growth 
and development of Federal funding programs designed to support 
and augment various educational endeavors on the state and local 
levels. In the wake of these funding programs (particularly 
since the Elementary and Secondary Education Act of 1965) there 
has come increasing concern that these support programs actually 
accomplish the objective of improving American Education (how* 
ever this may be defined). Thereforei more and more attention 
has been paid to the principle of accountability in the expen- 
diture of Federal funds, and to the "evaluation" of Federal 
support programs. 

There have of course been many responses to the pressure 

to evaluate. One of the most significant ones has been the 

recent establishment of the Joint Federal/State Task Force on 

Evaluation consisting of representatives of the U.S. Office 

of Education and of the Chief Stave School Officers of 20 of 

the States. Ihis Task Force (sometimes called the Belmont 

Croup after the place where the organisational meeting was held) 

signed an Agreement* under which it is attempting to develop a 

Joint Comprehensive Evaluation System (JCES) through which to 

* The Belmont agreement is shown as Appendix A to this report 
and the twenty Belmont States are listed as Appendix B 
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meet the Increasing demands for evaluation In connection with 
Federal/State funding programs In education. It la the nature 
and characteristics of this JCES with which the present study 
Is concerned. 

It is Important to recognise at the outset that one of 
the distinctive features of the JCES Is Its concern with the 
evaluation of the (management of the) funding process, as well 
as with the Impacts of the substantive programs being funded, 
that this dual concern Is essential becomes obvious when the 
nature of the Federal support system Is analysed. It can 
readily be seen that to a very large degree, the Federal support 
system Is a "delivery system" designed to transmit the funds 
appropriated by Congress to State and local levels where they 
are translated Into educational programs. Thus, the JCES must 
in large measure consider the adequacy of the funding process . 
and Its managemen t. as well as the Impact of the consequent 
educational programs on schools, teachers, pupils, etc. 
Furthermore, it Is In this respect that the JCES becomes a 
valuable management tool, as well as an assessment system, for 
it is well-known that good management demands a constant flow 
of feedback information regarding the effect and impact of 
current management practices. 

With the above background, and full appreciation of the 
importance assigned to "management" in the JCES concept, It is 
appropriate to discuss In more detail below the basic concepts, 
needs, and purposes underlying the development of the JCES. 
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As the JCES Is particularly concerned with evaluation as It 
relates to management, a goodly portion of the discussion 
centers on defiling the relationships between the JCES and 
management considerations. 

In a sense, development of the JCES can be viewed as 
an extension of the normal management review and evaluation 
practices necessary to the efficient management and control 
of large organizations and distribution chains. Everyday 
good management practice demands that the managers at each 
level be Informed of both the effectiveness of their own 
staff procedures and the effects that their actions may Save 
on the end products of the organization. The development of 
the JCES Is In reality the refinement and redesign of a set 
of existing evaluation requirements originally derived from 
legislation. These early requirements had been developed In 
haste at the beginnings of Federal support to educational 
development, and sjbsequent events have shown them to be not 
entirely adequate to the management review task. Thus, the 
development of the JCES should be considered an experimental 
development of a needed management tool, particularly with 
respect to the evaluation of the funding processes and of the 
effects of these processes on the target groups. 

This view of evaluation Is based on the recognition that 
management consists primarily of two major functions: planning 
and execution, but that each functional cot of the manager must 
be based on his knowledge of the results of his prior planning 
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and execution and Its success or failure. In this light, the 
evaluation of effect Is a necessary step to provide feedback 
to managers about the adequacy of their decisions, plans and 
acts. 

Some Problems and Needs 

The recent proliferation of Federal funding for educational 
programs at State and Local Levels has resulted In a rather 
unwieldy management system across the three levels of management 
responsibility. It has also led to a recognised, but generally 
unsatisfied, need for an adequate evaluation methodology to as* 
sees the processes and products of the various educational pro* 
grams developed. Bach of the Federal Acts and Titles author- 
ising either general or special funds for educational support 
has also called for either periodic or continuous evaluation 
of the resulting Programs. While legal requirements for eval- 
uation and reporting arc usually not specific, they ire at 
least Implicit In all Titles. The Implementation of these 
educational grant Programs and the development of substantive 
programs in States and Local agencies has frequently been done 
hurriedly. Developers were usually operating under time cons 
stralnts which Imposed expedient rather than optimal management 
and evaluation procedures for this installation. 

This may be Illustrated by the vertical (by Program or 
Title) management structures which developed within USOB and 
wete soon paralleled In most of the State and local agencies. 
Such vertical structuring of the organisational components of 
an agency can be very efficient for getting something done, 
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such as the rapid dispensation of large sums of money in this 
case. But it can also result in a large degree of overlap, 
with parallel personnel performing the same functions and 
handling the same or similar information and data; frequently, 
the counterpart persons may not even be aware of the duplica- 
tion. 

Such duplication has apparently occurred throughout the 
Fcderal/State/Local management chain, at least to the extent 
that many separate Programs and their (sometimes) redundant 
personnel are required to report highly similar Information to 
higher levels in the form of program- related, fiscal, or other 
reports. This is best illustrated by the current requirement 
for approximately 123 separate statistical reports imposed on 
the State and Local agencies reporting on the various Programs. 
The exact degree of content redundancy is unknown but the 
overlap of report titles would indicate considerable content 
overlap, also. Regardless of actual content redundancy, the 
multiple requirements post a heavy burden for clerical and 
managerial personnel at all ^vels in the management chain. 

Multiple reporting las ^osec other problems aho. It has 
resulted in some lacks of cross-program information required 
for efficient management of the overall Funding and Program 
development process. For example, precise information is 
lacking Jtroo many Programs and agencies in thtee specific areas 
crucial to effective Program management: 

1) VSat are the specific needs for support in State and 
Local areas? 




2 ) 

3) 
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What Is the amount of money required? and; 

What educational projects and programs work best 
under which conditions? 

These data are frequently lacking at all three major management 

levels and this lack Is likely to be as detrimental at the local 

agency level as at the Federal level. 

The Joint Comprehensive Evaluation System is directed 
toward meeting these information needs and evaluative require* 
ments at all three levels. Further, the system is intended to 
provide a vehicle for obtaining project/pupll oriented data, 
as veil as management oriented data. This will allow an adequate 
basis for evaluation of both the processes of the Fedeml/State/ 
Local "delivery system" for educational support, and the 
products of the Projects Instituted through the Federal/State 
support programs. 

Purposes of the JOES 

There are three related, broad purposes being supported 
by the JCES development. These are: 

1) To consolidate and reduce th a proliferation of 
statistical and evaluative reports which have 
been generated under current requirements; 

2) To develop a more management-oriented evaluation 
capability, together with a product-oriented evalu- 
ation capability, to permit more effective program 
planning and more efficient management of Program 
Funding disbursement and Substantive Program planning; 
and, 

3) To develop and geminate materials and techniques 
for study shi v\ >ing related to local staff devel- 
opment to assise in high quality, consistent, Local 
evaluation efforts. 
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JCJj S Benefits 

The current evaluation system development is unique in 
that it may be the first time that parties from Federal and 
State levels have Joined together to examine(and to do some- 
thing about) the problems surrounding the management of dis- 
bursement of Federal funds. Here, for the first time. State 
personnel are becoming involved in defining the processes 
whereby both the needs of their States and the effects of 
programs within their States are to be assessed, This Joint 
effort provides the States the opportunity to participate to 
some degree in the Federal planning and simultaneously should 
provide them a better view of the true extent of needs for 
both specific funding supports and for management improvements 
within the Federal/State/ T ocal allocation and disbursement 
system. 

In addition the implementation of the JCES should result 
in a number of more tangible benefits for Local, State and 
Federal agencies. The System implementation should: 

1) provide a better assessment of the needs of pupils, 
schools, Districts and States; 

2) provide a better capability for the estimation of 
the Program strategies necessary to meet these 
defined needs; 

3) provide indications of the levels of funding required 
to implement the strategies; 

4) provide some indication of the mix of funds (by 
mandated uses) 

5) improve grants management at both Federal and State 
agency levels; 
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6) aid determination of the nature and extent of 
technical assistance required at the local levels; and 

7) increase the focus on training requirements for 
evaluators and program managers at the State and Local 
levels. 

Thus the JCES implementation should become an important 
management tool for Agency Managers, Program Managers and 
Grants Managers at each management level through provision of 
a capability for near- continuous monitoring of the grants 
management and program developments. This monitoring and the 
proposed direct feedback to all consumers can provide Federal, 
State, and local administrators with up-to-date, progrum- 
relevent, assessments of program processes and the sequential 
progress that results. 
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THE JOINT COMPREHENSIVE EVALUATION SYSTEM 

This section presents a description of the development and the 
current status of the JCES and its elements. The first part of this 
section formulates an important distinction which appears necessary to 
the discussion of the JCES. This is the identification of the operating 
"Delivery System" and its discrimination from the developing evaluation 
system. While the evaluation system will be a tool for the management 
portions of the Delivery System, the '^Delivery System" itself is the 
overall system concerned with the transmittal of funds from funding 
sources through to substantive programs. A description of the Delivery 
System, and its operating objectives and goals is presented below. This 
description is done at a quite gross level but should provide sufficient 
clarification for the discussion of the evaluation system development. 
The Delivery System 

The Federal/State/Local Delivery System for funds distribution 
and translation of dollars into substantive projects is illustrated in 
Exhibit 1. This diagram sketches the relationships between the several 
management levels in the flow of dollars through the Funding Subsystem 
into and through the Substantive Subsystem to achieve an impact on the 
schools, teachers, and pupils. The Funding Subsystem may be considered 
to be almost entriely a fiscal management chain, which deals with the 
problems of grants applications, approvals, dollars, and fiscal account- 
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ability. This management chain operates at each level according to 
predefined rules to pass down the Federal monies and to fund specific 
programs or projects in either State or Local or school end-points. 

Some funds also go toward augmentation of on-going activities, so not 
all funding is related to establishment of new projects in schools. 

The Substantive Subsystem is concerned with the type and level 
of activities which are needed, or are to be activated with the funds 
received, at the State, District, or school levels. Persons in this 
subsystem must be capable of defining and developing educational 
programs, curricula, and activities, and should be capable of perform- 
ing the evaluative tasks necessary to assess their success or failure. 

This subsystem may be thought of as functionally separate from the 
Funding subsystem; however, the personnel performing the two sets of 
functions at many levels are the same persons or at least the same 
types. Frequently there is another type of overlap, in that the 
functions required of one person may simultaneously affect both sub- 
systems* Conceptually, however, these are two distinct subsystems of 
the Delivery System, and, on the basis of the differences in skills 
apparently required for fiscal and for educational decisions, it would 
appear most desirable for different persons to operate in the two 
systematic functions most of the time. 

An important concept for evaluation inherent in this distinction 

' ' \ 
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is that either of these two subsystems could conceivably be operating 
with a high degree of efficiency (within some constraints) even if the 
other subsystem were not functioning well at all. That is to say, while 
the Substantive Program subsystem could certainly be limited in innovative 
or augmented program development without funding, it might still accom- 
plish its functions even if the funding management were operating 
inef f iciently. Similarly, the Funding distribution subsystem could be 
operating smoothly and efficiently within a State and Local chain, but 
the conversion of funds to substantive programs could be quite ineffici- 
ent and ineffective at the same time. 

This major distinction is drawn to underscore the importance 
of process measures of the systems operation in addition to the product 
measures associated with student changes. Without the process type of 
measures, such as those related to the efficiency of operation of the 
Funding Subsystem, erroneous conclusions could be drawn from product 
measures like Pupil achievement scores alone. Process measures allow 
one to examine the how and why of a successful or unsuccessful program 
as well as simply the what happened , indicated by the terminal results. 
This is especially important in view of the partial interdependence of 
the subsystems in the Delivery System to be evaluated here. (The 
implications of the distinctions between process and product evaluation 
will be fully developed as a part of Phase III of the current effort.) 
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The JCES Goals and Development Approach 

Development of systematic evaluation procedures, Instruments, 
and techniques to assess the effectiveness of the Delivery System des- 
cribed above requires detailed consideration of both the Delivery System 
goals and objectives, and the basic objectives of the JCES itself. That 
is, the Delivery System is designed to perform certain functions at some 
desired level of adequacy and these can be thought of as the system goals 
or objectives. Similarly, the evaluation system is designed to answer 
certain defined questions about the system performance and to report 
these answers to designated users; these can be thought of as the JCES 
objectives. The development of goals is considered below. Additional 
developmental considerations persented in this section include, the 
developmental approaches to be applied, and the basic evaluation model 
which describes the JCES development. 

Delivery System Goals 

Prior to evaluation of any system the goals or objectives of that 
system must be specified in sufficient detail to allow development of 
criteria or this step has not yet been completed for the Federal/State/ 
Local Delivery System, and the development of these criteria will require 
a somewhat lengthly and detailed process of defining system and program 
objectives in measureable terms. For the moment, the general legislative 
goals must be accepted as sufficient goal statements. These statements 
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reduce to a set of requirements imposed on the management structure of 
the Delivery System that demand the production of "good" results (better 
educational opportunities, and more effective educational programs for 
the educationally deprived, whether these persons are disadvantaged, 
socially deprived, or otherwise partially incapacitated in their freedom 
to achieve a "good" education*)' The legislative goals could be stated 
at much greater length, but they boil down to about the above statement: 
and, as is readily observable, are not of much help in defining precise 
goals and objectives for the Delivery System's operations. These require 
much more precise and detailed specification prior to the eventual 
evaluation of the performance of the system, Such a specification can 
and must be done early in the subsequent development* It is only against 
such definitive objective statements for the system that its actual 
effectiveness and efficiency may be justly evaluated, This specification 
is expecially critical for the management aspects of the Delivery System, 
since this part of the system is the least definable in terms of per- 
formance objectives, 

JCES Objectives 

The evaluation purposes of the Joint Task Force are to assess, 
and to relate to intended util ization, the effects of the various Funding 
and Substantive Programs resultant from the various legislative Acts and 
Titles with which the JCES is to be concerned (See Exhibit 2). The 
evaluation system will not evaluate the adequacy of the legislation per se, 
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EXHIBIT 2 

Listing of the Legislative Acts and Titles of Programs 
to be evaluated under the JCES# 

1. ESEA Title I: 

j a. Low Income Groups 

b. Neglected and Delinquent Children 

c. Migrants 

t * 

2. ESEA Titles II 

3. ESEA Title III 

4. ESEA Title V, Section 503 - Flow Through Funds 

5. ESEA Title VII 

* ' V 

6. ESEA Tltile VIII 

7. ND# Title III 

8. NDEA Title V-A. 

9. Civil Rights Act, Title IV 

10. Follow-Through 

The above Acts and Titles fund Programs which are under the supervision 
of the Bureaus of Elementary and Secondary Education and of Adult, Vocat- 
ional and Technical Education* 

The Vocational Education Amendments Act may be added to the set of Titles/ 
l Programs to be evaluated at a later date* 

# * Appendix C presents a detailed review of the evaluation requirements 

and authorizations of each of these Legislative Acts and Titles. 
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although such might result, serendipitously, from the evaluation of 
effects. Rather, the system must provide for appropriate inputs to 
legislative and administrative thinking through reports of progress and 
effectiveness as well as of needs still unsatisfied by current programs. 
Thus, the evalution is directed at the effects of the legislation, and 
toward possible modification, but not at the legislation, itself. 

Effects of legislation and the delivery of funds to States and 
local agencies are comprised of several types. Among these are: 

1) Specific, unique, and novel management requirements 
laid on the Federal, State and local agencies, requiring additional 
expenditures of time, manpower (perhaps uniquely unavailable skills), 
and Federal funds for administrative chores, or overhead tasks. 

2) Specific substantive program developments instituted 
within local agencies, schools and classrooms - these have specific 
effects on the individual pupils and on the total classes, which may be 
on academic skills, social development, or other student characteristics. 

3) Other non-specific effects of the dollar inputs to local 
agencies which may or may not be reflected in specific programs directly 
affecting the individual children in the school area. 

Other individual types of effects could be identified but these seem to 
be representative of both the complexity of the problem and the two general 
types of effects: Management effects, or Process-related effects of 
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Programs; and, Pupil /pro Ject effects in schools and districts, or 
Product-related effects. The evaluation must be able to deal with these 
two types of effects of Programs wherever they may occur. 

On the basis of the above, the goals of the Joint Comprehensive 
Evaluation System are to provide the capability for accurate assessment 
of school and pupil educational achievement; to evaluate the differential 
effectiveness of the various Federally funded Programs on the basis of the 
educational and related results of the projects instituted; and to 
evaluate the process of funds distribution through the Delivery System. 

The planned evaluation system will provide this capability for multiple 
effectiveness evaluation and will directly assess the results of the 
diverse projects, activities and treatments in the schools of the Nation. 

The evaluation system will also investigate the degree to whi< h 
various needs of the students, local agencies and schools, are being met 
through current Federal contributions in conjunction with the local and 
State financing. This goal includes identification and assessment of the 
needs not being met as well as of the degree to which current programs 
are effective for recognized needs. 

The goals of the JCES may be restated as the following objectives, 
toward which the instrument development and the planned implementation 
are directed (Note that the descriptive material Is relevant to the 
evaluation of process , as well as background for product evaluation): 
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1) Describe individual school characteristics and pupil character- 
istics at the school level; 

2) Describe the projects, activities, or treatments in which 
pupils are participating at the school level; 

3) Relate school and pupil characteristics $o specific projects, 
activities and treatments; 

4) Describe overall Program progress; 

5) Identify successful projects, activities or treatments; and, 

6) Assess needs in terms of: 

a) flow of services, b) populations being served, c) projects, 
activities, or treatments uot being provided, and d) pupil 
changes not occurring. 

In order to assess the attainment of these goals, the components 
of the JCES must collect the necessary data. In ^n effort to determine 
whether or not this data will be collected a set of first level evaluation 
questions has been identified, These questions wfll be compared against 
the various system components to determine if all the required information 
has been considered. The evaluation questions themselves have been 
abstracted from a set of policy questions provided by OE and represent 
the types of questions asked by OE program managers, (For measurement 
purpose such questions must be refined, A beginning and further discuss* 
ion are given in Appendix D, The evaluation questions are: 



- 19 - 

1. What are the background characteristics of the general 
student population at the school, district and state level? 

2. '‘-’hat are the background characteristics of the students 
who participate in OE programs anc of those who should part- 
icipate and don't? 

1. What are the characteristics of the schools and school 
facilities that participate and that do not participate In 
the O.E. program? 

4. What types of School personnel - from administrator to 
teacher - participate In O.E. programs? 

5. WSat are the various types of general services provided 
to the school, school personnel and students? 

6. How are the various programs administered at the state, 
district and local levels? 

7. What are the costs of these programs and what are the 
various sources of funds available to cover then? 

8. What are the special programs with services directed at 
well defined target groups or selected populations? 

9. How effective are these programs In meeting the needs o«‘ 
the groups they are Intended to serve? 

10. What are the Identifiable benefits from these programs 
to the school, school personnel and students? 
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11. h v ha t is the extent and results of program research and 
development activity? 

12. How do the schools .Interact with the community and local 
government agencies to solve community problems? 

It should be remembered that these evaluation questions are 
offered at this time as being representative but not exhaustive of those 
which might be asked* cutting across specific programs* and emphasizing 
the kinds of questions which may be asked of the JCES in the future. 

Basic Developmental Approach « 

The basic approach taken in the development and implementation 
of the JCES is founded on two major points: * 

1) Obtaining close cooperation of the States* and thereby the 
Districts* through a Joint development effort directed toward providing 
evaluation and lessened workload payoffs to both the States and Districts* 

* Certain practical and political considerations have influenced 
the approach taken in the development of the JCES. These included a 
variety of items which might be construed to be obstacles or hindrances 
to the development and installation of any comprehensive evaluation 
system • particularly were such a system to Impose new work loads on 
the contributing agencies in the States and Districts. They also Include 
a number of legal and Internal regulations which tended to restrict the 
time limits for application and completion of the evaluation system. 

These considerations are discussed later in this report. 
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as well /a to the USOE; and, 

2) Designing and developing the data collection subsystem so 
as to consolidate and reduce the existing evaluation reporting requirements 
of local agencies, by emphasizing the interventions of existing data- 
ct reams and minimizing the requirements for new information from these 
agencies. 

The first point above led to the development of a Joint Federal/ 
State Task Force (the Belmont Group) under a cooperative working agreement 
directed toward the development of a comprehensive and consolidated 
evaluation system. Such a Joint group appeared to be the optimal found- 
ation for the development of a comprehensive evaluation system fully 
responsive to the needs of both Federal and State and local evaluation 
interests. Furthermore, such a working group was deemed highly desirable 
from the view of effectiveness in attaining both the cooperation and the 
information essential to the successful development of the JOBS. 

The second point dictated a review of all existing data streams 
and other locally available data to evaluate the suitability of these 
data for the overall comprehensive evaluation purposes. Some of the 
current data sources available (or replacement) by the JCLS are listed 
in Exhibit 3. These data sources, especially the multiple, separate ones 
related to individual Funding Program evaluation based on currently man* 
dated reporting requirements, proved to be highly redundant. Thus, the 
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EXHIBIT 3 

Some current Data Sources available for 
Conversion or Replacement under the JCES. 

1. Grants Applications Forms 

2. Project Evaluation Reports 

3. Fiscal Reports 

4. Statistical Reports 

5. Program Evaluation Reports 

6. National Evaluation Studies 

a. ESEA Title t 

b. ESEA Title II 

c. NDEA Title III 



d. NDFS Title V-A. 

7. Educational Audits 

8. Progress Reports 
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major effort was defined as the specification and selection of those 
data streams which could be successfully Intervened for the purposes 
of the comprehensive evaluation of all Federal programs and the necessary 
modification of the instruments, etc,, related to each data stream, to 
meet the precise needs of the JCES, 

The formation of a Joint group to define and implement the JCES 
i 8 believed likely to be highly effective in reducing many of the State 
and local problems previously encountered in Federal evaluation efforts. 
Since State and local agencies participate in the design and Implementation 
planning of the effort, this joint group also has advantages for the States 
etc,, since evaluation data will be rapidly fed back to the local agen- 
cies to aid them in their management and educational program development 
functions, It is anticipated that the joint effort will result in 
considerable payoff to all concerned in terms of reduced workloads, 
reduced external reporting requirements, and more accurate and more 
comprehensive evaluation of both management and educational activities 
at all levels of the Delivery System 1 * operations. 

Evaluation Models 

A major function of the developing JCES 19 to evaluate the 
effectiveness of the various Federally sponsored programs which have 
been introduced into the school environment. As with most evaluation 
schemes, th*s system is based on the observation and assessment of 
changes in the target populations over time and the evaluation of the 
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programs on this basis. This is essentially a longitudinal change model 
for evaluation and is based partially on the assumption that the treatments 
(programs and activities) inlorduced will, if they are effective! produce 
a set of desirable, objectively measureable, changes in the target 
populations. Such as an assumption is the basis of all change study 
evaluations. 

An additional dimension of evaluation is required, however, 
centering on the management aspect of the Federal Programs and their 
implementation within the States a.)d Localities# This evaluation require- 
ment derives from the necessity for all managers at all levels to be 
informed about the adequacy of their own operations in order to modify 
them toward improved functioning. This, requires evaluative data 
collection, analysis and feedback of data and reports to Agency Managers, 
Program Administrators and Grants Managers at all involved levels. 

The same longitudinal change model postulated above as necessary 
for substantive f rogram evaluation can be used to assess management funct- 
ions also through the direct incorporation of process variable measure- 
ment and feedback to the concerned managers. However, evaluation of the 
adequacy of performance of these managerial functions requires the 
development of specific standards or criteria for such task performances, 
and these ate mostly still lacking within the operating system, Therefore, 
such criterion development »ust be a part of the evolution of the 
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comprehensive system. It appears that such criterion development should 
be undertaken by tho combined forces of the Federal/State/local agencies 
and peraonnel who would bn most affected by their application. Certainly, 
those criteria must be both performance oriented and pragmatically sound 
in relation to the specific management taska and functions performed. 

Exhibit 4 presents an Illustration of the goal model approach to 
evaluation in this context. Thia approach characterized the funding and 
substantive program development and management as the overall Input and 
Process activities within a measurement paradigm designed to identify 
the adequacy of performance of these activities on the basis of relation- 
ships between specified Process and Outcome (dependent) variables. The 
approacn assumes the definition of the specific process variables ass- 
ociated with a given Program development and implementation (including 
its management) and the capability for associating measures of these 
with criteria for their performance and also with the Outcome variables. 

The Outcome varlablea msut also be specified and related criterion 
values In order for the goal model approach to be fully effective in 
aasessment of the valuea or benefits derived from a given program activity. 
For individual programs and for comparisons between Programs, the 
appropriate analysis and evaluation of the variables suggested by this 
paradigm and the changes In each aet over time, through a sequential 
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analysis, would allow the correlation of process variable changes with 
outcome variable changes. Such analyses could provide the basis for 
assessment of both management and implementation performance in relation 
to the products of these programs - the changes in the student population. 
Such evaluation would provide at least partial answers to questions 
concerning whether or not specific programs work and also which of 
several related but different program implementation approaches work 
better than others . 

The extension of this goal model approach to multiple programs 
would be termed the systems model approach, and an abbreviated paradlfm 
of such comparisons is illustrated in Bchibit 5. This shows the situ- 
ation which cojld exist given adequate measurement of the Inputs/Processes/ 
Programs and Income and Outcome variables for a set of parallel Program 
implementations - either between schools, or districts, or States, or 
even conceivably between Federal funding programs and management tech- 
niques as applied to different programs and activities. The multivariate 
comparisons which could be drawn between the Program and process variables 
in relation to the outcome variables, as well as simply those between 
processes and between Outcomes, separately, would allow the evaluators 
to draw conclusions related to the overall adequacy of the different 
programs on each of several bases - including management, implementation, 
in-school projects or processes, and pupil changes. 
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The above model defines four types of variables, which are 
listed, with some examples of each, in Inhibit 6, These types are: 
Population variables ! relating to the definition of the characteristics 
of the incoming students for any given program or project in a school or 
district; Input /Process /Program Variables » relating to the characteristics 
of the managment and implementation of the specific funding and substan- 
tive programs which are impacting on the given incoming population in 
the school; Intermediate variables , which define the set of "internal" 
characteristics of the students which may or not be modified by the 
programs and, if modified, may or not affect measures of the fourth 
category; the Dependent Variables . which can be though of as the ultimate 
outcome measures and Include the overt behaviors of the students and are, 
In general, those characteristics of the student population which are 
intended to be modified by the educational support programs. 

The concepts and models described above are believed to be 
adequate to meet the essential evaluation requirements posed by the 
Federal /State/Local Delivery System. It is further believed that the 
longitudinal change model applied to the comparative evaluation of both 
outcomes and input /process variables will actually allow formulation of 
the comparative judgements about different programs and processes, of both 
management and operation of programs, which are necessary to the Iterative 
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development of a well managed and effective system of educational support 
programs . 
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EVALUATION SYSTEM AND SYSTEM ELEMENTS 

The conceptual development of the evaluation system in relation to 
a simplified depiction of the Delivery System is illustrated in Exhibit 
7. This shows the Evaluation system functioning in an overlying 
monitoring and controlling relationship to the total Devi ivory System, 
as it must in order to perform its major functions related to the total 
management tasks. This Exhibit also indicates the eventual relationships 
between the Delivery System, the Evaluation system, and the developing 
management plans of USOE and the State and Local Agencies. As shown, 
the outputs of the evaluation system, both internal and external are 
intended to provide data and reports feedback directly to the State 
and Local Agencies as well as to the Federal managers and evaluation 
personnel. This feedback, together with the potential for detailed 
interpretation of data and incorporation of these into the overall 
management plans, can be used to aid the gradual evolution of the State 
and Local management plans and capabilities. Whether this will operate 
through direct outputs to the State and Local Agencies, as indicated 
above, or through the Chief State School Officer (CSSO's) association 
with the developing Belmont organization is incidental to the actual 
system development, but such involvement could develop through either 
route and probably will include both. 

The major management loop implicit in the overall system development 
and its extensions is that leading to the raanagment plan developments 
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mentioned above. The provision of data to the USOE/CSSO management 
personnel will contribute to decisions and development of policy and 
position statements, program memos and related evaluative and management 
products directed toward the States and local districts, or to Congress, 
as warranted by the data indications drawn from the system. These 
USOE/CSSO policy developments and position papers will also be inputs 
to the development of State and Local management plans. 

These management personnel, responsible for the crucial decisions 
about Funding Programs, Grants Applications and Federal Policy are 
critically important internal users of these evaluation system data. 

It is assumed that these personnel can have access to all data sheets 
and internal and external reports routinely generated by the System 
and the associated personnel. They should also have the capability to 
make specific inquiries of the system data base and to institute new 
searches or analyses of data to obtain data needed for decision making 
but not anticipated in the original design of routine output capabilities. 
These decision makers and their currently identified needs, must play a 
large part in the initial development of the evaluation system and its 
data analysis plan through definition of their information requirements. 
The JCES data system must be designed, explicitly, to provide the infor- 
mation and data needed by these personnel. 

Exhibit 7 also indicates roughly the kinds of reports which will 
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be output from the system and some of the potential users of these 
reports* Without detailing these for now (See System Reporting Require- 
ment, below), they will consist primarily of either data sheets, 
containing essentially raw or summarized data, or finished, interpreted 
reports of analyses related to either Programs or to Target Groups, Since 
the major output of the system will include the mandated Program reports, 
the external reporting requirements include the dissemination of these 
and similar reports to Congress, to the Educational Research Information 
Center (ERIC) and to other external consumers. 

This exhibit also indicates the development of a User Interface 

portion of the evaluation system. Whether this be a manual or man-machine 

interface (assuming part of the evaluation system will be computerized) 

is undefined at this point; the importance of recognition of this inter- 

* 

face lies in its relationship to the provision of data sheets and reports 
generated for specific purposes. This interface supplies the various 
users - both internal and external to USOE - with the required data out- 
puts to answer either routine or special questions based on the data a- 
vai lability and the data analysis system. 

Total System A p proach 

The development and implementation of the JCES as described above 
is intended to follow a total systems approach; that is, the elements of 
the system are intended to follow an integrated development and applica- 
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tion plan and the analysis and evaluation of the resultant dara will be 
based on a comprehensive data analysis plan, relating all of the data 
from all instruments. 

The total system can be throught of as accomplishing two major 
tasks and supporting a third activity, external to the system itself but 
related to the continuing evaluative development of the Delivery System. 
These are: 

1) The system will directly accomplish the reduction and con- 
solidation of the various statistical; fiscal and evaluative data 
input requirements of management evaluation for the Federal, State, 

and local agencies; and, it will be designed to provide direct feedback 
reports on a hierarchical basis, and on-demand, to personnel operating 
at each cf these levels. These personnel served include Agency Managers, 
Program Administrators, and Grants Managers at each level, as well as 
eventually the Congressional and public information needs associated 
with this area. 

2) The system will allow direct assessment and evaluation 

of that subset of educational projects included within the sampled groups 
(those actually evaluated with the different instruments) and an extrap- 
olation, on the basis of the samples, to the entire population of projects. 
Ihis assessment and evaluation can include the assessment of which kinds 
of projects work best under what conditions and for which kinds of target 
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pupils, as well as the identification of the additional needs among 
target groups for educational support of various kinds. 

3) Secondarily, through the accomplishment of the evaluative 
functions above, the system will contribute directly to the continuing 
modification and improvement of management activities at all levels 
of the delivery system. This contribution to the better interactive 
management of the Funding Programs and translation of these into educa- 
tional projects can only accrue under conditions of mutual Federal/State/ 
Local recognition, acceptance and dedicated application of the infor- 
mation to be derived from the satisfactory performance of the above 
two functions. This ancicipated outcome of system development and 
implementation is more dependent upon external "people factors" than 
on the specific development of the JCES, however, this aim is important 
to the development and specific implementation of the system elements 
and to the report generation capability, in particular. These must be 
designed to facilitate the mutually interactive utilization of the data 
derived and processed by the system. 

Development of the system necessary to meet these functional needs 
is likely to require an iterative cycle of continuing refinement due to 
changing recognition of arising needs in both the management and project 
evaluative areas. Modif ica tions of vailable data usually result in the 
evolution of novel desires for utilization which may cause new data 
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collection or analysis needs, or simply the generation of new reporting 
formats. In any case the current intent is to design the total system 
and to implement it to the degree possible and to allow it to evolve 
to meet such new needs as may be defined later. The current design 
calls for an integrated set of instruments aimed at obtaining statistical 
financial, student, and project data and a complete data analysis system 
to provide the interrelated analyses necessary to provide the summary 
and evaluative outputs required for effective management and evaluation 
practices to be instituted at each level of the Delivery System. Each 
element of this system will serve a definite purpose and will not general 
ly be redundant with other instruments. Some redundancy is probably un- 
avoidable in survey and data collection instruments to be administered 
at separate times in the same levels, schools, etc., but this is to be 
reduced to the bare minimum consistent with the demands for both data 
identification and data consistency checks. 

The expansion of the evaluation system design shown in Exhibit 8 
indicates the set of instruments currently under development and the 
relationship of these to the the three major data collection and analysis 
functions which can be identified as? the General Statistical Data 
Subsystem; the Management Evaluation Subsystem; and, the Project/Pupil 
Centered Evaluation Subsystem. Instruments comprising these subsystems 
are described below with repect to purposes, data types and applications. 
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Descriptions here are general, but Appendix E presents detailed descrip- 
tions of all instruments together with an item-by-i tem indication of the 
required data. 

The General Statistical Data Subsystem 

This is not really a new development but rather an incorporation 
into the system of two existing NCES instruments and files to serve pri- 
marily as reference sources and as descriptions of the population of 
Districts, schools and pupils. These instruments are the Elementary and 
Secondary Education General Information Survey, (ELSEGIS) and the 
Universe File. Together they comprise the basis for general information 
development about the total population with respect to some very limited 
population and numbers data. 

1) ELSEGIS - This instrument is a survey instrument used to 
assess the existing school systems and pupils of the country and its 
emphasis is on collection of these data about all schools instead of on 
those involved in Programs under ESEA or NDEA, as other instruments 
sometimes do. It collects data on: the number of schools in local 

systems by organizational level, grade, and size of system; enrollment 
and pupil /teacher ratios in local public school systems by account and 
size of system. These data are collected by a mail survey of a sample 
of public school Districts developed and implemented by NCES. 

The degree to which this instrument will be incorporated into the 
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total JCES is not yet fixed. The data collected partially overlap with 
that from the CPIR (discussed later) and this may indicate that the 
redundancy should be eliminated. But decisions on these points must 
await detailed examination of both sampling and data analysis plans 
which are not yet complete for either instrument. In any case, the 
ELSEGIS data would be the basis for routine statistical summaries similar 
to those already prepared. These summaries could be useful to the man- 
agers and decisions makers at all levels, so whether the instrument and 
its raw data are incorporated in to the JCES or not, it is likely that 
the end-products of the survey will be incorporated into the output 
system of the JCES. 

2. Universe File - This is essentially a canvassing of all 
schools in the country and the determination of their enrollment by 
grade and the instructional staff they maintain. These data are 
collected through a mail survey completed at the District level. The 
major purpose of the File resulting has been, and will continue 
to be, its utility as a sampling frame for sample generation for other 
survey applications. 

The Universe File Data will certainly be incorporated into the JCES 
development for the sampling purposes. These data would probably be 
maintained in NCES as they are currently, but the availability as a 
sampling frame would be used in the design and implementation of the 
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various other JCES instruments. As indicated above the outputs of 
the general data subsystem will at least be used in the JCES by selected 
internal and external consumers, but it is not expected that any novel 
reporting will be generated from these instruments. The single possi- 
bility for such unique analysis and reporting would be the cross relation 
of these data with those from other instruments, and this would be only 
possible at a general level because of probable lack of sample duplication. 
The Management Evaluation Subsystem 

This subsystem contains the set of instruments directed at obtaining 
the basic data required to evaluate the major management functions at 
each level of the management chain. There are three instruments which 
are to be developed and applied to this evaluation purpose. These are 
the Program Reference File (also important to the Pupil/Project 
Evaluation process), the Consolidated Program Information Report (CPIR), 
and the still embryonic State Education Agency Management Evaluation Sur- 
vey (SEA-MES). These are outlined below. 

1. Program Reference File - This instrument will create 
a permanent file, to be updated annually, that will contain information 
on all schools and school districts in the 20 State Belmont Group. In 
addition to the name and address of each school and its school districts, 
the following data will be stored in the file: 

Name of District Superintendent 



ERIC 

‘hiWIfBiiffBfitiiiaiailM 



A3 - 



Name and Telephone Number of Person Responsible for Federal 

Programs 

Grade Span of School 

School Membership 

Degree of Urbanism in School Attendance Area 

Listing of Federally Supported Aid Programs in Which Schools 

Participate by Grade level. 

This instrument and the File will be partially based on the Universe 
File in that the initial generation of the basic identification data 
will be used to canvass the Belmont State Schools for the additional 
data required. Once these data are available, they will comprise a 
needed universe record of the programs active in schools and Districts 
for the Belmont States. As such this File will serve as the basic 
sampling frame for development of spatial State oriented samples and 
Belmont aggregate samples for the special studies of these States using 
the major instruments of the JCES and perhaps some supplementary instru* 
ments as might be developed in conjunction with the individual States 
(or all States . ) 

2. Consolidated Program Information Report (CPIR) - Ihe 
CP1R is designed to serve three broad purposes: a) to permit Slate and 

Federal program officers to determine the extent to which Federal/State 
programs and services reach pupils and schools as intended! b) to assess 
the broad elements of program effectiveness and efficiency at the local 
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district level; and c) to satisfy Federal statistical reporting require- 
ments for Federal Funds. The CPIR will replace many of the 123 statis- 
tical reports that vere previously required by USOE. In replacing these 
various reporting requirements, the CPIR and its central analysis, together 
with the capability for feedback to all management levels, will provide 
the capability for direct output of suninary and raw data to LEAs, and to 
SEAs as well, which have been previously unavailable. It is presumed 
that these data availabilities will increase the capability of Agency 
and Grants Managers to monitor and improve the management activities 
for which they are responsible. 

Among the data that will be collected by this instrument utci 

a) Dollars expended by source of funding 

b) Services and programs provided by these funds 

c) Identification of the number of children by larget group 

needing services and number benefiting from the programs and 

services 

d) Staffing patterns by programs and services 

e) In-service education by source of funding 

The instrument will for the first time provide a coordinated 
look at the various Federal funding programs impacting on local school 
districts. The 19 o 9 CPIR has been sent out to the school districts and 



the 1970 fora is presently in preparation. 



- 45 - 



3* State Educational Agency Management Evaluation Survey 
(SEA-MES) - Although In the planning and development stage currently, 
the SEA-MES will be an Instrument (or more than one) designed to collect 
Information required to evaluate the State and local agency management 
of Federal funding accounts and grants for specific projects. It Is 
Intended that the Ins trument (s) will replace many of the specific fis- 
cal accounting reports currently required from Local and State agencies 
through Incorporation of these separate reports Into a consolidated 
Instrument set. Application of some of these instruments may be as 
often as quarterly but the major reporting will probably be summarized 
in an annual documentation* 

At the moment, this Is In the initial planning stage with a work- 
ing draft planned for presentation at the second quarterly Belmont 
meeting sometime in July 1970. The instruments are viewed as question- 
naires somewhat similar in design to the Consolidated Program Information 
Report. This evaluation wilL be limited to details about the management 
of the program. For example, they will deal with areas such as: 

What programs do you administer! 

What types of people assist you in managing these projects? 

Hov many of each type? 

What funding levels are dealt with, etc.? 

The information accumulated will then he analyzed and used to evaluate 
how effectively the program was managed. 
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As a separate part of the SEA-MES .the questions of "needs assess- 
ment" will probably be addressed. Tills would be a continuation of 
current assessment requirements, since me part of the ESBA Title III 
program, the 50 states were alloted funds during FY 1 68 and FY‘69 to 
m^ke an assessment of each of their needs in the area of education. 

These assessments have been completed and reports huve been received 
which Identify the educational needs of each state. In addition, the 
states were asked to determine which of the identified educational needs 
are critical and to determine those for which Title III might effective- 
ly demonstrate Innovative and exemplary programs. However continuing 
re-assessment through the medium of the SEA-MES would assist Federal 
and local managers In fund allocation decisions. The previous report 
data could also be Incorporated into the JCES, If desirable. These 
could then be analysed by 0E to summarise the most critical needs of 
the Nation to determine a strategy for the Title III program In relation 
to other Programs. 

These Instruments and the data collected will fora the basis for 
multiple specific outputs of the evaluation system. As separate fiscal 
accounting requirements for the USOB and/or Utate or Local Agencies 
develop, separate reports may be specified to meet these needs. However, 
an additional and major aspect of these data will be their utilisation 
in relation to Project/Pupil evaluation data (see below) to provide 
Indications of benefits /costs ratios for different programs and projects. 
Thus, these management data will provide the basis for direct management 
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evaluation by managers at all levels and will simultaneously allow the 
more effective evaluation of the product aspects of the Substantive 
Program Subsystem of the Delivery System* 

The Pupll/Project Centered Evaluation Subsystem 

This subsystem forms the heart of the product evaluation capability 
of the JC&3 since it is the source of all specific data about pupilSi 
their progress, and the specific activities which have been provided, 
either under Federal, State or Local funding programs. The instruments 
to be described below will allow data collection concerning specific 
Districts, schools, teachers and pupils in addition to the development 
of descriptions and evaluations of local projects which have been imple- 
mented under diverse funding programs. These data can be analyzed to an- 
swer the crucial questions relating to project effectiveness in the 
field and what kinds of projects work best with which target groups 
under what kinds of conditions. 

1. Comprehensive Pupil* Cantered Instruments * This set 
consists of four parts designed to gather data on the school district, 
the school, the teachet. and the pupil. Part of the Pupil Centered 
Instruments will be aimed at determining the extent to which individual 
students participate in the various project and activities described 
through the Project Descriptor (See below)* In addition, the Tupil 
Centered Instruments will gather data on the background of the students 
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participating in the projects, and their school achievement. 

The School District Questionnaire collects the following types of 

da ta: 

a) General Information (including salary data and number of 

schools in the district) 

b) Test Data Information 

c) Parent Involvement Information 

d) Personnel Training Information 

The Principal Questionnaire collects data of the following types: 

a) General School Information (including location, membership, 

and attendance data) 

b) Instructional Organisation 

c) School Facilities 

d) Student Body Description 

The Teacher Questionnaire collects the following types of data: 

a) Teacher Background 

b) Class Characteristics and Organisation 

c) Teaching Method and Program of Instruction 

d) Teaching Concerns 

The Pupil Questionnaire collects data as follows*. 

General Information (including grade, sex, age, and absences) 
Pupil Background Characteristics 
Academic Program Participation 



a) 

b) 

c) 



O 

ERIC 



48 



d) Ancillary Service Participation 

e) Pupil Behavior 

f) Pupil Performance 

Only the Pupil Centered Instruments for Elementary Schools have 

been developed in draft form. The final version of these instruments 
will be submitted to NCES and the Bureau of the Budget for clearance 
on January 16, VJbrk has not yet begun on the secondary school instru- 
ment . 

The Pupil/Project Centered Evaluation Subsystem will depend primarily 
on the set of data generated through the four Instruments described above 
since these are the only instruments which are likely to be Implemented 
throughout the Belmont States at this time. However, there are three 
other somewhat related instruments under development which may be used 
in various ways in conjunction with the JCES to augment the data avail* 
able to the users of the system. These three instilments are considered 
as part of the Project /Pupil Evaluation approach and thus are described 
briefly below. 

2. Comoon Status Measures • These consist of two tests, 

Basic Verbal Status and Occupational Cognisance, They were originally 
developed at two levela for application to National samples of children 
at the 4th and 11th grades. Currently, these measures are viewed as the 
introduction of a new National survey, and the desire to avoid the bur- 
dens of new surveys suggest! that these instruments may find only limited 
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use In special evaluations of special programs or groups, or might 
be welcomed into some or all of the Belmont States on a local survey 
basis . 

The Basic Verbal Status measure has 12 questions* The skills being 
measured are vocabulary recognition and reading comprehension. In no 
case is the student required to make any generalizations or inferences 
about the selection. The Occupational Cognizance measure is designed 
to assess the student % knowledge of occupations and his occupational- 
educational expectations. This measure will have 12 questions, the 
first ten of which will test the pupil's knowledge of: 

a) the education or training required for specified occupations 

b) the nature of work involved in specified occupations 

c) the relationship of other occupations to a specific occupa- 
tion 

d) the recognition of the field of work corresponding to a 
specified occupation* 

The last two questions ask about the student's own occupational and edu- 
cational expectations* The Common Status Measures are presently in 
draft form and both the elementary and secondary school forms are 
currently being pre-tested* 

3, Project Descriptor Questionnaire - This instrument is 
based on a taxonomy of projects, programs, and activities. It will pro- 
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vide Information on the various types of projects and programs operating 
In the schools. For each project the following types of Information will 
be gathered: 

a) Type of Project or Activity 

b) Source of Funding 

c) Duration of Project 

d) Number and Background Characteristics of Participants 

e) Organization for Instruction Including the Teaching Methods 
Bnployed, Subject matter, and Characteristics of Cognizant 
Personnel 

f) Facilities, Equipment and Materials Utilized 

8) Project administration (including planning, evaluation, and 
dissemination) 

The analysis of the deta from the project descriptor will provide 
detailed Information relative to the services provided through each Fed* 
eral legislative title. For example, reports will Indicate the structure, 
methodology, materials, equipment, and personnel used In projects In com- 
pensatory reading In secondary schools. Because the descriptor will In* 
elude Information In the types of pupils participating In the various 
projects, analyses will report the kinds of services being rendered to var* 
lous kinds of pupils within States. When the entire system Is lmple* 



mented, the relative effectiveness of various kinds of services for vatl- 
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ous kinds of pupils will be reported. Program managers In State agencies 
and at the U» S. Office of Education will thus have comprehensive pictures 
of the kinds of services each of the Federal programs Is generating. 

4. The Local Evaluation Handbook (AIR Guide) - This Is a 
manual which Is designed to assist local and State personnel In the 
preparation of local project evaluation reports In standardised format. 

It contains guidelines for the content and preparation at the level of 
narrative reports on Federally-funded projects. These reports parallel 
the Information obtained at the school level by the Project Descriptor, 
which was developed separately from the (Xilde. In order to further 
standardise these reports and bring them In line with the data collected 
from the Project Descriptor, It has been recommended that the Project 
Descriptor Taxonomy be Included as an appendix to the AIR Guide and used 
In conjunction with It. 

The National application of these two new Instruments, the Project 
Descriptor and the Local Evaluation Handbook, faces the same problems 
discussed under the Common Status Measures, above. However, they hold 
much promise for development of an Important set of data about local 
projects and their Implementation that Is not generated by any other 
current source. On this basis It has been suggested that these Instru- 
ments be Implemented through the Federal /State/Local management chain on 
an Informal basis, probably starting through the Belmont Organisation and 
perhaps extending eventually to other States. 
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It is not anticipated that this approach would be universally 
successful, in that many local systems and school might be unable or 
reluctant to participate. However, even a small and non- representative 

I 

group of such coordinated and standardize project evaluations, developed 
by local personnel, would be useful to all levels of managers in the 
Management chain. Such evaluations could lead to a much clearer pic- 
ture of what is working, and where, and how. Such data are simply not 
available now, and they could be of much use to Program Administrators 
and Grants Managers in determining the substantive suitability of new 
project applications. They could also be disseminated widely to Local 
administrators and be useful in informing these personnel of what has 
worked and what has been only marginal in other locations and applications 
Such information might assist the more effective application of local 
as well as Federal and State funds in the Districts. With limited 
funding (a constant local problem) informed decisions about such alloca- 
tion are required to improve effectiveness. And for these purposes the 
information from other project developments and their evaluations by 
the local personnel could be Invaluable. 

Relationships of JCES to Btlatltm Evaluation Activities 

The relationship of the JCES development to the ELSBG1S and Universe 
File of the existing activities has been described above. However 
additional ties and substitution relationships also exist. Some of these 
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are briefly described here. The JCES does not represent the first 
effort to evaluate the functioning and effectiveness of Federal education* 
al support programs. Evaluation efforts have been carried out In the 
past In connection with ESEA Titles I, II, III, and NDEA Titles III 
and V-A*. The frequency of these studies has not been annual, but 
some have been carried out more than once. In addition to these, 
there have of course been a great many state and local studies attempting 
to evaluate the Impact of Federal programs In education. Further, the 
fiscal accounting system of legislation, requires a continuing series of 
financial reports both quarterly and annual. 

It should also be noted that the JCES effort pertains, at this 
time, only to the 20 Belmont states. Thus, It Is legitimate to a6k 
the extent to which the JCES duplicates or parallels an existing evalu- 
ation system. The answer to this question Is "not at all." There is 
really no present "system" In any systematic sense except for the fin- 
ancial and narrative formatted reports requred by USOB. It la planned 
that these evaluation efforts will be consolidated Into the various 
subsystems oi: the JCES. Two parts of the JCES, the Consolidated Program 
Information Report and the Blcmentary Pupil Centered Instruments will 

* As noted previously, a review of all Legislative Titles which will 
be evaluated by the JCES is presented In Appendix C to this report. 

(Motet page $4 blank) 
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be extended to all 50 of the states in the near future, and it is hoped 
that the remainder of the JCES will be ready and accepted into the 50 
states by 1973. In fact, much of the present evaluation effort will be 
ef tively subsumed under the new system, since the 1969 Survey of 
./■ ' * or/ Education (Title I, ESEA) forms the basis for the new 
I/O Pupil Centered Instruments (covering all Titles) and will be ex- 
tended in 1970 to all 50 states. 

The Sy3tem will handle program information sources such as: 1) 
Application reports, 2) Fiscal reports, 3) Project Evaluation reports, 

4) Statistical reports, 5) Program Evaluation reports, 6) The previous 
national evaluation studies referred to above, 7) Educational audits, 
and 8) Program reports. With respect to the CPIR, the 60 million or so 
response items contained in these reports, requiring some 32 man years 
per year to punch and edit, will be reduced on the CPIR to about 4 
million items needing only 8 man-years to punch/edit. 

Thus, the components of the JCES are designed ultimately to replace 
the many separate studies and reports for OE funded elementary and second 
dary programs. For example: 

1. The CPIR replaces existing local statistical report for 
ESEA Titles I, II, III, NEA Titles III, V-A, and the neglected 
and delinquent statistical report under ESEA Title I. 
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2. The School /Pupil Instrutoents replace separate national 
surveys in ESEA Titles I, II, III, and NEA Titles III, V-A. 

3. The Project Descriptor Instruments and the AIR Guide 
ultimately may be used to standardize local project reports 
mandated for ESEA Titles I, III, VII and VIII* 

4. The SEA Management Report ultimately may replace mandated 
State statistical and evaluation reports for some titles of 
both ESEA and NDEA. 

Comprehensiveness and Redundancy Analysis 

As part of the detailed study of the developing JCES, a series of 
analytic examinations of the various elements of the JCES were con- 
ducted. These included analyses of the comprehensiveness of the 
elements of the system with respect to the basic evaluation questions 
asked by Program Managers, and analysis of relationships of element 
coverage to the currently required reporting schedules for all legis- 
lative Titles, and an examination of the item-by-item redundancy 
among the system instruments. The results of these analyses are re- 
ported below under the headings of Comprehensiveness and Redundancy. 

Some discussion of potential modifications of elements and their 
applications follows these reports. 

Comprehensiveness of JCES Elements 

A major question in examining an attempt to evaluate anything 
is: "Does evaluation approach actually cover and evaluate every- 

thing it is supposed to?" The question asks whether the system is 
comprehensive of the evaluation needs with respect to the subject 
of evaluation. This question has been examined for the JCES 
elements through two approaches: first, the coverage of the JCES 
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elements in relation to the basic evaluation questions posed by 
Program Managers in U30E was reviewed; second, the relationship 
of the JCES elements to the current USOE reporting requirements 
(from States and Local Agencies) was reviewed. These two exami- 
nations are reported below. 

Evaluation Questions and JCES Elements - A series of 
twelve evaluation questions were formulated from the sets of 
policy and evaluation questions posed by USOE Program Managers. 

These were presented earlier as part of the Section on JCES, Goals 
and Development. These questions were abstracted from the several 
highly similar se s of policy questions prepared by the Programs 
Managers of the various Titles within the various Bureaus of USOE. 

The question examined was whether or not the current JCES elements 
provide coverage of each of these formalized questions. Exhibit 9 
presents the matrix of JCES elements set against the 12 Evaluation 
Questions. (The questions are abbreviated for exhibit purposes.) 
Within the body of the matrix, a check has been placed at the 
intersection of an evaluation question with a JCES instrument if 
that instrument provides some (not necessarily all) of the data 
required to provide an answer to the specific questions represented 
by the short titles shown herein. It should be noted that only 
the major intents and the main sections of each instrument were 
considered in relation to the questions at this time. An item- 
by-item analysis of these relationships has been partially completed 
This preliminary analysis indicates that the JCES elements 
can provide most of the kinds of data required to answer the ques- 
tions posed for response by the Program Managers. The matrix 
also shows that several different elements of the JCES can be 
expected to provide some data in relation to most of the questions. 
Although this does not necessarily imply redundancy of the separate 
instruments, this initial analysis and examination of the general 
contents of the individual elements led to the conduct of an item-by- 
item analysis of the overlap across instruments. 
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The results of this redundancy analysis are reported later in this 
section of the report. 

Reporting Requirements and JCES Elements . --An 
analysis similar to that presented above vas performed to evaluate 
the comprehensiveness of the JCES elements with respect to the 
current legislative requirements for reporting. These were derived 
from a table provided by USOE/PPE and were examined against the gen- 
eral content and major sections of the various instruments, As i i 
the above examination of the JCES instrument coverage, a matrix 
was constructed and checks were entered at the intersections of the 
individual elements and a report title to which it was believed 
the JCES instrument pertained. This matrix is presented as Exhibit 
10 . 

Two facts should be pointed out here. First is the fact that 
this examination was performed without; benefit of any detailed informa- 
tion on what the listed reporting requirements were supposed to be 
comprised of in the way of data. The comparison that was made in 
creating this matrix was between the Ti tie of the report and the 
general content and major data indications drawn from the JCES 
instruments. Thus, there may be many wrong inferences in this 
matrix, simply because they were made on the basis of the report 
titles alone. The second major fact to be made clear is that this 
comparison has been made on the basis of equally putative inferences 
with respect to the contents and coverage of the to-be-developed 
SEA-MES, the State management evaluation survey device. This device 
is in the planning stage only, with no draft instrument or outlines 
available for examination. These two facts combine to render the 
comparisons shown in Exhibit 10 very tentative. Nonetheless, the 
Exhibit provides a useful broad picture of the coverage of the system. 
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COMPREHENSIVE EVALUATION SYSTEMS ELEMENTS 
AS RELATED TO CURRENT BESE/BVTE REPORTING REQUIREMENTS 



BESE PROGRAM REPORT 
REQUIREMENTS 



ESEA 89-10 as amended Title I 
Education Deprived Children in 
low income areas 

l.SEA Evaluation of Title I 
Programs (Admin) OE 4320 
2.CPIR - OE 4484 
3. Survey of Compensatory 
Education 

OE 4434,4434-1,^434-2, 
4434-3 

4. SEA Admin. Expenditures & 
LEA Program Expenditures 
OE-4319 

5. Quarterly Report on Dis- 
tribution of funds to LEAs 
OE-4384 

6. AIRS Project descriptions 
(proposed) 

7. List of ESEA Title I 
LEA - OE 2320 



ESEA 89-10 Title I 
Neglected and Delinquent 
children In Institutions 

1* Evaluation of N&D Program 
(SEA, State Agencies & In- 
stitutions) OE 4426 
2 *CPIR - OE 4484 
3.Stat» Report from State In- 
stitutions 0E 4375-1 
4. State Agencies Expenditures 
for N & D Program Expendi- 
tures OE 4319 
5. Annual Survey of Institu- 
tions for Neg. & Del. 
children - 4376 
6. Summary of Approved Pro- 
jects OE 4453 
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4. Evaluation OE Report 
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contract) 

5. CPIR - OE 4484 

6 , Expend! ture Report on LEA 
Program Expenditures 
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School Library Resources 
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7.CPIR - OE 4484 
8. SEA Statistical OE 4462 
9. SEA Financial Report 
OE 4462-4 
10. LEA Proposed Budget Summary 
Expenditures Report 
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ESEA 90-247 Title VII 
Bilingual Education 



4. Erd of Project Report: 
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4. Enl of Project Report: 

Stat OE 4480-1 Eval OE 4480 
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Activities OE 4480-6 
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2. Formal - 0E 4480 
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Continuation 
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CRA 88-352 Title IV 

Equal Educational Opportunities 



2. Annual Evaluation 

3. CPIR - OE 4484 

4. Final Fiscal Report 
OE 1115 

5. Fiscal Operations OE 1115 
(Semi-annually) 
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REQUIREMENTS 
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Vocational Education 
Title I 

Part A - State Advisory 
Councils 

Part B - Programs for Stu- 
dents with Special Needs - 
Basic Grants to States 
Part D - Innovation 
Part B - Residential Voc. 
School 

Part F - Consumer & Home- 
making 

Part G - Cooperative Voc. 
Educa. Programs 
Part H - Work Study Programs 
Part I - Curriculum Develop- 
ment 

Title III, Planning 6 Evalua- 
tion. 




2. Descriptive Rpt. of Voc. Ed 
Program Act. OB 4130 
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5. Enrollment 6 Completion In 
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6. No. of Teachers in Voc., 
Ed. OE 3136 

7. Status of Teachers OE 
OE 3136 

8. No. of State & Local Admin 
Personnel OE 3135 

9. Financial State of Voc. 

Ed. Funds OE 3129 

10. Expenditures by Source 
OE 3130 

11. Expenditures by Level 
OE 3131 

12. Expenditures by Purpose 
OE 3132 

13. Projected Status & Esti- 
mated Expenditures 

OB 3133 
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89-4 amended by 90-103 
Appalachian Vocational Educa. 
Construction Grants & Supple- 
mentary Construction Grants 
(Reports Unspecified) 



.State Plan - 
(States follow 
provisions of 
their state plan 
under PL 88-210) 



81-920 Civil Defense Education 



2. Financial and Inventory 
Rpt. 08 3047-1,2,3 6. 3c 

3. Monthly Activity Report 
0B 5126 



1. State Plans for 
Participation 
(Unnumbered) 



MDTA 87-415 as amended 
Manpower Develop. 6 Training 
Title II On the Job Training 
Title II B - Inst. Training 
Title II C - Redevelopment 
Areas 



1. State Training Plan 
OB 3117 & 0B 3117 

2. Training FroJ. Info 
08 3055 

3. Project & Budget 
Approv. 0B 3123 
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4. Expenditure Fpt. for 
Stp » Direction & Super- 
v on OE 3056 

5. Cost; of occupational 
Training OE 4000 

6. Federal excess personnel 
property available for 
transfer OE 3126 
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Application 
Proposals /Level 



? 



7 



? 



7. Request for Inventory ? 

Adjustment OE 3127 

8. Auxiliary List Amendment ? 

OE 3128 



ESEA 89-750, 90-247,90-576 
Title III 

Adult Basic Education 
Grants to States 
Special Projects 
Teacher Education 



1. State Plan 0E 3051 

2. Fiscal Estimates 
OB 3052 

3. Application for 
Grant OB 3121, 
3121-1 & 2 



5. Survey of Participants in 
Adult Basic Education 
0B 3081 

6. Teacher Training Partici- 
pants Pom 0B 3120-3 

7. Annual Financial 0B 3059 
3. Annual Expenditure 

0B 3119 

9. Annual Expenditure 
0B 3120-6 

10.<^iarterly Program Rpt. 

• LB 3091 



■ 7 



? 



4. Project Applica- 
tion OB 3120, 
3120-1 & 2. 



7 7 

7 7 



7 7 

7 7 
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On the basis of this extremely preliminary analysis, it 
appears that the large majority of the current reporting requirements 
of the States to USOE can be included (with some degree of compre- 
hensiveness) in reports to be generated by completion and analysis 
of the data entries required within the JCES elements. This is 
actually based on the assumption that the SEA-MES will in fact 
cover all State management reporting requirements as these were 
listed in the original table. 

JCES Element Redundancy 

Following the two analyses of element coverage reported above, 
an item-by-item analysis of each of the JCES element instruments 
w*s performed. The items were then inspected for inter-element 
redundancy. (The item-by-itun listing of elements is included 
as part of Appendix E.) Although a superficial look 

at the instruments might lead one to expect many fully redundant 
item s, a more detailed examination shows that the situation is more 
complicated than that. The analysis showed that although questions 
way be worded in exactly the same manner on two or more instruments, 
the reference groups to which they are directed differ. Examples 
of this situation are discussed below. 

Funding * The Project Descriptor Instrument (PD1) asks 
for the amount of federal funds "requested" and "received" for the 
current fiscal year by Title. Since the PDI is filled out at the 
School District level for each specific project separately, the 
amount of funds relates to one project only. This prqject may be 
in operation in one grade level, several grade levels, one school, 
several schools, all schools, etc. The Pupil Centered Instrument 
(PCI) District Questionnaire askeJ about the amount of funds 
"approved" for current fiscal year by Title. This item may sound the 
same as the PDI item, but since this item of the PCI is filled 
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out at the District level, the amount of funds relates to all pro- 
jects in the District under a specific Title. The (Consolidated 
Program Information Report (CPIR) which is also answered at the 
District level, asks about the amount of funds "expended 11 (for the 
former school year) for various types of services and activities 
by Title in each District. This is a finer breakdown than was 
called for on the PCI as it reports funds by type of service or 
activity. From this, it can be seen that these items are not 
strictly redundant. Additionally, the questions ask for funds 
requested, received, approved, and expended. These terms are 
certainly not entirely redundant, although the PDI's "received" 
might correspond to the PCI's "approved" in some cases. 

Impact Area - The PDI asks for the characteristics of the 
population in an impact area for a specific, project. Data on 
variables such as per capita income, unemployment rate, total 
population, urbanism, and ethnic or racial group membership are 
collected. The PCI School Principal Questionnaire asks about the 
characteristics of pupils and their families in each school, 
specifically minority group membership, public welfare recipients, 
parents' education and urbanism of school’s location. The PCI 
Pupil Questionnaire also asks about pupil characteristics. This 
questionnaire contains items relating to minority group member- 
ship, public welfare recipient, language spoken in the home, employ- 
ment status of parents and family income. Even though the«e three 
Instruments collect the same type of data, they are not redundant 
because they are applied to different sets of people (i.e. • project 
participants, school, population, and class.) Data from the PCI 
Pepil Questionnaire cannot be swed to yield accurate school 
information because or.ly a 3mall number of pupils ave sampled 
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In each school. Finally, the PD1 data and PCI School Principal 
Questionnaire data do not overlap because schools and projects do 
not always overlap. 

Number of Schools - Both the Pupil Centered Instrument 
School District Questionnaire and ELSEGIS collect data on the 
number of public schools in a school district by grade span. The 
PCI School District Questionnaire asks for the number of schools 
in each of the following categories: PRE-K, K-3, 4-6, and 7-12. 

ELSEGIS asks for a finer breakdown, but the information is largely 
redundant. 

School Attendance - The CPIR collects data on public 
school membership for districts for each grade level from Pre- 
Kindergarten to Grade 12. ELSEGIS collects data on pupil 
enrollment in school district for nursery school, kindergarten, 
elementary, and secondary. This can be derived from the CPIR. The 
PCI School Principal Questionnaire asks for the average daily 
attendance and school membership for Kindergarten and Grades 2, 

4, and 6. The PCI Teacher Questionnaire asks the teacher to 
indicate the number of pupils in his class. There is some redundancy 
here betwee.i the PCI Principal and PCI Teacher Questionnaire 
since the Teacher Questionnaire is administered to all 2nd, 4th, 
and 6th grade teachers in each sampled school. 

Staffing * ELSEGIS asks for the number of instructional 
staff by type and the number of classroom teachers by highest 
level of education completed in each district. The Project 
Descriptor Instrument collects data on the number of personnel by 
type for each project. This is not a redundancy since the data 
is collected for two non-equivalent groups. 



Instructional Approaches * Both the PDI and the PCI Teacher 
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Questionnaire ask about approaches to the presentation of instructional 
materials* i.e., whether a topic, subject matter, unit, skills, or 
activity centered approach is used. In the PDI, the teacher is 
asked to describe the individual project with respect to the 
above, while in the PCI Teacher Questionnaire, the teacher is asked 
to describe the approach used in math, reading, and language. The 
question is not exactly redundant since the question 1a asked 
regarding two different groups that cannot be equated. 

In- service^4rai ciing - The PCI Questionnaire collects data 
on the amount of fuO<Je and number of personnel by type that are 
participating in In-Service training programs in each school 
district. The CPIR collects this same information for each 
district. This is perhaps the only truly redundant question with 
respect to both type of information and reference group. 

The questions cited above are the only ones in the JCES 
System that have any redundancy at all. It is clear that not very 
many items could be eliminated from the present instruments on the 
basis of redundancy alone without a resultant loss of information 
necessary according to the original designs for the system 
development and implementation. 

Summa r y of El^pient Anayses 

On the basis of the findings of the reviews reported above, it 
appears that the JCES are, in general, both relatively comprehensive 
in meeting the general evaluation requirements and relatively non* 
redundant with respect to specific item coverage. 

Certain reservations must be placed on this statement, however. 
First, with respect to comprehensiveness, there is a serious lack 
in the instrument set at this time because of the non-exLs tence of 
the SBA*MES. This Instrument is intended to proviJe evaluation data 
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with respect to the management chain functions within the States 
(and perhaps local) agencies. Since it does not yet exist, 
the statement of relative comprehensiveness is based on the 
assumption that the SEA-MES can be developed as a comprehensive 
instrument which will fulfill its purpose in a satisfactory and 
complete manner. 

Second, also with respect to comprehensiveness of the 
instruments, an item-by-item appraisal with respect to the evalua- 
tion needs is still needed. The estimation of comprehensive 
coverage of an instrument in relation to the evaluation questions 
and the reporting requirements were made on the basis of an over-all 
review of the instruments and their contents. The actual instru- 
ments were used in this estimation process, but not on an item by 

item basis. Therefore, it can not be said that the instruments 

are totally complete with respect to every possible question which 
should be asked, However, it may never be possible to say this 
about any set of evaluation instruments designed for any purpose, 
and the JOES instruments currently available appear "comprehensive 
enough * 1 to accomplish a large part of their massive job. 

On the question of redundancy, an item by item check and 

comparison was made across all instruments. Some redundancies were 
discovered, but only a few. Other items which had initially 
appeared redundant from instrument to instrument were generally 
asking for about the same information, but were requesting it from 
different levels in the management chain of the Delivery System. 
Thus, little real redundancy exists across the currently 
developed ins trtments , and it Is not thought to be a serious problem. 
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SYSTEM REPORTING REQUIREMENTS 

This section presents the results of current analyses of the report- 
ing requirements and the data analysis developments necessary to support 
the developing JCES. Included herein are discussions of the major 
functions to be served by the information processing portion of the 
evaluation system 9 the consumers and their utilization of the JCES and 
its data, the reporting requirements for consumers, some examples of 
various report types vhMh might result from the developing report 
generation systx*m» and a preliminary view of the data analysis plan 
necessary to develop the detailed reporting requirements and the specific 
data sets necessary to provide the outputs. 

System Functions and System Reports 

Each agency in the management chain has several major functions 
which must be carried out and reported on: It must function as a 

grantor : as a monitor of projects funded under the f rants; and as an 
evaluator of the effectiveness of the program. Each agency must also function as 
a reporter of the effects to higher levels to the Congress, and to the 
general public. 

As (an example of) a grantor * &ESE make funds available to States and 
local school systems. Funds can be made available in a general non-categoti- 
cal format or with categorical restrictions on utilization. In the former 
case, attempt is made to assess the impact of the additional money on the 
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program and the granting agency (SAFA Is a good example) uses relatively 
simple criteria to determine the needs of the local education agency. 

Under other Tltles/Programs, the agencies are required to assess the 
equity of the allocation of grants. Under some title, monies must go 
to specific area of educational need according to the level of need 
(Title I grants to local education agencies Is a good example). In 
other cases, specific target populations or types of programs must 
receive the aid. 

The JCES will serve these needs by collecting statistical Informa- 
tion concerning the areas of relative educational need through the 
CPIR and SEA-MES. The resulting Information will be used to determine 
bolih the equltablllty of grants allocations and the budgetary require- 
ments for the future. 

Each agency must monitor Its activities also and must collect 
data on Its day to day performance whether at USOB, SEA, LEA, or local 
school levels. These data must be related to the management effectiveness 
of personnel within the agencies as well as to information on program 
effectiveness In the substantive programs taking place In the schools. 

Both management performance reports and program performance reports 
should be available and should be related to other statistical Information 
In the JCES. Field reports of monitors shou’d be Included also as a 
retrievable part of JCES. These would Include reports of visits to 
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various education agencies to corroborate reports of management 
effectiveness submitted to the JCES. 

JCES Is also designed to assist each agency's operation as an 
evaluating agency. Several major areas of evaluation can be Identified. 

The first of these has to do with the direct observation of the operating 
agency, itself. This is viewed as a self-evaluation of management 
effectiveness. Thus, the JCES must provide a tool for Program Managers, 
Administrators, Agency Chiefs, and Grants Managers to use in examining 
their own performance, Identifying areas of need, and defining remedial 
action. A second kind of evaluation concerns the evaluation of each 
stage of the delivery system and Involves regional Interests, state 
education agencies, local education agencies, and frequently, sub-dlvislons 
within the local education agencies having program decision responsibili- 
ties but not engaged In educational services. Development of an adequate 
evsluatlve pattern for this complex delivery system will be a major 
goal of the present effort. 

Finally, each agency must serve as an evaluator of substantive programs 
carried on In schools by real people working with real children. The 
task Is difficult but the broad outlines of evaluation of this form of 
activity have been laid down. Information Is collected concerning the 
relative level of skill, abilities, etc. of the recipients of the service; 
a specified treatment or activity is carried out: the subsequent levels of 
the skills and abilities are related t-^ the eatller measurement; and 
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the program Is "measured" In terms of the Improvement or change which 
occurs In the recipients. Information for reports of this type must al- 
so be collected In JCES. 

System Output 

Output Users 

On the basis of the above designation of the Delivery System and 
JCES interrelationships, the following is a list of the most likely 
consumers of the JCES output reports: 

1) Congressmen, who want to know what the money has been 
spent for and what effects have resulted; 

2) Agency Managers at all levels: Federal, State, and Local 

- who want to know what their agencies have accomplished and how well it 
has been done; 

3) Grants Manager - again, at all levels - who want to know 
just how the money has been disbursed, for what and under what conditions 
it is being applied; 

4) Program Administrators - at all levels - who want to know 
what has been done within both funding and substantive programs, and what 
effects tne efforts have had on the intended target groups; 

3) Research and Evaluation Personnel - both within And outside 
the Federal /State/Local chain - who want to know what is going on, who is 
being served, by what kinds of programs and projects, and under what 
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conditions; also are concerned with the effectiveness of which programs 
in terras of what level of effects can be observed under which of the 
combinations of initial conditions and Project applications; 

6) State and Local level Adrainstrators (CSSO's and Superin- 
tendents), who want to know how well their State or Local schools and 
students are doing in relation to other States, etc., and in relation to 
expenditures under the various Federal and State funding programs; 

7) The educational and general public, who are interested 
in the educational progress in schools and also in the relative costs 
expended to achieve given degrees of excellence, as well as in the questions 
as to which kinds of educational programs actually produce results in 

the schools. 

Probably there are other potential users of the system outputs, 
but this partial list will suffice for the moment, since it indicates 
a broad scope of interests and a broad set of outputs required to meet 
these interests. 

It should be pointed out that the major users both in number and 
frequency of usage will be the management and evaluation personnel 
within each level agency related to the JCES. This means that the major 
outputs of the system will be those required by the Agency Managers, the 
Program Adminitfrators , the Grants Managers and the Research and Evaluation 
personnel at each level of the system^ operation. Although all of the 
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is these latter persons wo roust be satisfied in the initial development 
of the reporting subsystem of the JCES, It appears that the reporting 
requirements of these personnel can be satisfied through a combination of 
statistical (descriptive), fiscal accounting, and evaluative reports 
centering on the program processes and the products of the substantive 
projects in the schools. Some characteristics of the report types 
these personnel will require are discussed next# 

System Reports 

When the work "report" is used about a management activity such as 
the Delivery System (from Congress, through the Secretary and Commissioners 
of BESE and hence to the SEAs, the LEAs, and individual school programs 
or projects) we must define our meaning carefully. Reports of varying 
levels of specificity and purpose range from visitation reports from 
monitors to annual reports published by the Secretary of HEW, They can 
vary in form from a single sheet of figures entered in a quarterly 
fiscal report to a comprehensive source of detailed fiscal data, such 
as the SAFA Annual Report. 

A complication here is shift in the perception of a report which 
occurs when it leaves the reporting agency aid travels to the receiving 
agency. Materials traveliug upward in the system are throught of as 
reports by the initiating agency. But, the receiving agencies may con- 
sider these to be items of data to be entered into their own report 
generating system. That is, they view incoming reports not as reports 
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at all, but as "statistical bits" to be entered into their files, 

This same shift in the meaning of the work report can take place 
within U30E. For example, incoming information from the field Is used 
to create a file from which a "report" of grants approved, children 
served, or monies expended can be generated. However, once this "first 
stage report" comes inco being it, in turn, becomes a resource unit in 
a larger process: that of generating annual reports or dissemination 

documents. These aspects of "reports" roust be considered in the further 
development of the report generating system within the JCES. 

Report Type s 

The reports to be generated by the JCES can be categorized along 
several conceptual dimension*. For example, there will be reports which 
serve che needs of individuals or small groups of persons and other 
reports which will be widely circulated to large number of persons, 

Report can also be categorized along the dimension of numerical 
versus narrative. Some reports will consist almost entirely of statis- 
tical and fiscal data organized in numerical tables with no significant 
amount of verbal explanation. Other reports will contain largely 
narrative and interpretive information and depend only in minor ways on 
the use of statistical or fiscal information. Finally, many reports will 
be hybrids, with considerable amounts of narrative or interpretive 
wilting to support and call attention to the statistical information. 
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Reports can also be categorized according to the purposes to which 
they will be put: administrative and management decision-making; report- 

ing to Congress and the public; development of legislative proposals end 
budgetary requests; dissemination of information concerning exemplary 
projects and effective approaches to local schools; etc. Some of these are 
described briefly below. 

Individual Feports: Feedback to States, etc. - The 
information available in the data base of the JCES can be made useful to 
individuals and small groups at various levels within the overall Delivery 
System. A significant aspect of the JCES plan is a cooperative relation- 
ship between the states and the Federal government. In order to reduce 
the number of reports flowing from the LEA$ and SEAs to the Federal 
level, a system of comprehensive reports is under development to 
significantly reduce the total number of forms required under the various 
programs at the local and state levels. At the same time, consideration 
is being given to the information needs of the States. Thus information 
generated at any level can be moved to higher levels where it can be 
merged and manipulated to provide the required information for Federal 
purposes. At the same time specific summary printouts by State! and/or 
District can be derived and returned to the States and Districts for 
their use; thereby avoiding duplicated effort in the collection and process- 
ing of information. Thus, a major function of the JCES reporting activity 
will le the provision of appropriate statistical printouts for the 
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individual States or Districts. Such "reports 1 * will be in the form ot in- 
dividualized data printout designed to meet the needs of single managers 
or evaluators or a small group of decision makers within a branch. 

Inquiry Concep t Another form of individualized report will 
be produced by the JOES is response to "one-of-a-kind 1 requests . An 
important part of development of the JCES capability is the development of 
a methoddlgy for prompt and appropriate information retrieval (or generation) 
"on demand." This information retrieval capability should not be limited 
to a simple "drawing account" for "statistics". The inquiry capability must 
provide for data manipulation well. 

The inquiry system will permit managers and administrators to inter- 
face with programmers and researchers to formulate specific questions 
that can be addressed to the JCES. These questions might be developed 
in response to special calls! legislative calls and budget calls received 
by the Bureau. A well designed inquiry system can also serve more general 
needs; it c n identify the frequently-asked questions and thereby perform 
two functions 

a) provide regular reports on these questions or issues. 

b) identify areas cf congressional ur Bureau concern and 
interest which warrant re-examination and the possible 
generation of legislative proposals. 

The proposed inquiry system represents a powerful Instrument, there- 
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fore. In the generation of special reports in response to special calls, 
budget calls, and legislative calls. 

Published Reports - Several major classifications of published 
reports can be identified. Some are periodic and repetitive in nature 
(the SAFA, NDEA and, more recently, the ESEA Survey Reports, are typical 
of these repeating or serial reports). Other reports grow out of the 
continuing interest of the Bureau in specific areas of educational 
activity or pract.ee. Some examples are: the Public School Finance 

Report, the Educational Directory of States (mandated, but format varies 
from year to year), the Handbook on Exemplary Practices, etc. These 
reports draw upon, but are not directly derivative from, the types of 
information to be available in the JCES. Taken together with the legis- 
lative review reports mentioned above, they can constitute the general sys- 
tem of reporting available at present. 

Anbther form of published report which may be generated from JCES 
statistics will be reports focused on the special target groups. 

Such reports would identify the multiple impact of various legislative 
titles on a specific group of persons (for example, migrants, dropouts, 
etc.). Such reports would also focus on educational sub-groups and report 
the impact of multiple programs. 

Report Specifications 

The following Exhibits (11a through llg) present some preliminary 
specification for some of the reports to be generated from data derived 
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EXHIBIT U-a 



SPECIFICATION FOR; 

USERS DATA SHEETS: MANAGEMENT 



Audience ; 

Program and Grants Managers In Federal, State and/or Local 
Agencies who need aggregated data concerning their level of 
Interest. OE-BESE will work through the Belmont Group to 
determine all persons or Job titles by whom such Information 
will be required* 

Authorship ; 

To be generated directly by JCES Data System On Request or as 
Routine Request. Specific data content will be defined by 
persons at each level of Interest. 

Content : 

These reports will be In the form of data printouts. They 
may merge data from the LEA's or the States to form state- 
wide or National counts, tallies and other cumulative Indices, 
as required by the level of Interest* 

Some cross- tabulation, or intar-action of variables Information 
will be available to managers through this feedback system. 
Statistics will more usually be actual counts, percentages and 
averages* 

(Analyses or data base queries of a more complex nature will 
be requested through the inquiry system. Frequently-requested 
inquiry items may then become parts of the regular feedback 
system. Infrequently requested data manipulations will be 
dropped from system*) 

Next Steps : 

Sit with Agency Managers, Program Administrators, Grants Mana- 
gers, Fiscal Officers of typical programs at each level to 
work out the format ard data contents most appropriate for 
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EXHIBIT ll-a(continued) 

title with which they are concerned. Concentrate on data 
needed for effective management as opposed to data required 
for published reports. Prepare Mock-ups of Printouts for 
evaluation and approval by division and branch managers. 
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EXHIBIT 11-b 



SPECIFICATION FOR: 

GRANTS MANAGEMENT INFORMATION REPORT 



Audience : 

SEM or USOE Grants Managers and Agency Managers, Congressional 
Committee Members, and Committee Counsel. Other persons Inter* 
ested In the general problem of disbursing funds through State 
Coordinating Agencies to local users of funds. The reports 
should indicate where bottlenecks are occurring, where unspent 
monies are accumulating, where "feast/ famine" conditions are 
occurring, or where cycles of fund disbursal are conflicting 
or out of phase with funding needs, etc. 

Authorship : 

Prepared (using Functional Users Data Sheets) by responsible 
SEM or USOE divisions and branches concerning their own grants 
management program. 

Content : 

Focus on Funding programs (perhaps across programs). 

* Amounts of money allocated (deliveries) 

* Funds remaining (inventory) 

* Grants and Proposals processed 

* Applications received and processed by SEA's 

* Evaluation of Application (sales) 

* Decision making speed 

* Re-allocation cycle 

* Management services provided 

* Personnel resource allocation problems 

* State and Local Management allocation problem 

areas 

Next Steps : 

Work with title coordinators to develop standard table of 
contents and report format. 
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EXHIBIT 11“ c 

SPECIFICATIONS FOR; 
PROGRAM ADMINISTRATION REPORT 



Audienc e; 

A report useful to managers at SEMS and USOE to share infor- 
mation on the goals, problems, thrust of each program; 
useful to Congress, especially • members of the House Education 
and Labor and the Senate Labor and Public Welfare Committees, 
in hearing from administrators how the intent of congress is 
being carried out, how loca? area needs are being met, what 
legislation is needed, etc, And useful to the educational 
publics (i.e#, NEA, NAVA, etc#) in helping them develop an 
understanding of the programs, policies, problems, etc. that 
are associated with the program. 

Separate Program Reports are to be prepared for each Title or 
related cluster of activities# 

Authorship ; 

Prepared by individuals responsible for Grants Management/Pro- 
gram Administration within a specific title area. Summaries 
of data as required can generated directly by the JCES Data 
System, 

C ontent ! 

Each report focuses on one Congressional Mandate and reports 
the degree of success of the manager of that program in meet- 
ing the intent of Congress. Each report presents JCES data 
which help to; 

Report implementation of programs 

Monitor conduct of programs 

Indicate program success 

Indicate project successes 

Spotlite weaknesses in current programs 

Indicate needs for new programs 

Show whether present funding levels are adequate 

Indicate if program is missing target population 

This report could bee* me a valuable resource during annual 
reporting to Congress and to the President. It serves as a 
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EXHIBIT 11-c (continued) 

This report could become a valuable resource during annual 
reporting to Congress and to the President, It serves as a 
vehicle for converting JCES data into legislative recommen- 
dations, support of continuing programs, and for presenting 
plans for reallocation of effort from one area to another, etc, 

Next Steps : 

Discuss published report needs with persons responsible for 
program administration and offices, prepare draft "table of 
contents" for sample report* 
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EXHIBIT 11-d 
SPECIFICATIONS FOR: 

MANAGEMENT PROCEDURES DISSEMINATION REPORT 



Audience : 

A report useful within OE as a device for defining management 
activities, methods, procedures and useful to SEA and LEA pro- 
gram managers to clarify how management information of the 
type generated by the LEA can be used at federal, state, and 
local levels to strengthen management; can be used for training 
and development of management and evaluating personnel at OE, 

SEA and LEA levels; and useful with Congress and the educational 
public to show how effective management principles are being 
"required" by tbe JCES. 



Content : 

This report (probably broken down into a series organized by 
program) would relate th* CESystem to the management technique 
or system. Some of the possible areas of focus would be: 

*• Paper flow and Documents used 

• Accounting: 

Cost 

Pupil 

• Procedures at: 

SEA levelo 
LEA levels 

• Training and Staff Development 

• Reporting Techniques (uses for data at SEA and 

Lea levels * school boards and public) 

' Efficiency Measurement 



These reports would also interpret to LEA and SEA managers 
the impact and uses to which they can put the JCES data for 
their state or local area. They would also serve as models 
for within-state paper-flow charting and encourage improved 
management planning at SEA and LEA levels. 
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EXHIBIT U-d (continued) 



Next Steps : 

Develop a clearer picture of the needs of member states in the 
way of Management Information and Development; prepare outlines 
for specific titles in the Management Information Report Series; 
work with existing materials showing paper flow and approval 
structures within the OE. 
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EXHIBIT 11-e 

SPECIFICATIONS FOR: 
EVALUATION REPORTS: TITLE- BY- TITLE 



Audience : 

Useful with Administrators, Congressman, as a device for 
interpreting the purpose, intent, goals, degree of success, 
types of approaches, etc* that are associated with the par- 
ticular TITLE reported upon; useful with project officers and 
professionals within LEA’s and at the school level to show them 
(by inference) how they compare with other projects under same 
title; shows them where they stand; what to aspire to, etc.; 
also for public dissemination and to ERIC. 

Content : 

These reports would provide specific, detailed statements of 
the costs/benefit associated with the Title Program being 
reported. Characteristics of typical programs; characteristics 
of unusually successful programs; and characteristics of unusual 
ly weak programs would be reported. 

These reports would focus on Title programs. They might be 
broken down along such lines as: 

• Recipients - percent reached; intensity of service 

pupil hours, etc. 

• Services - content, techniques, approaches, etc. 

• Materials - equipment, tests, community resources, 

etc . 

• Staffing - types of people, roles, man-hours, etc. 

• Costs - actual, attributed, etc. 

Some special problems associated with specific titles, would 
be covered in this series. An example is the problem of iden- 
tifying schools with exceptional educational need for Title I 
monies. The new JCES data may provide information for better 
distribution of funds. 



EXHIBIT li-e (continued) 



Next Steps ; 

1 • Prepare "Table of Contents" for each of a proposed list of 
"Title Evaluation Reports". 

2. Review concept and contents with OE title coordinators. 
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EXHIBIT 11-f 

SPECIFICATIONS FOR: 
EVALUATION REPORT: TARGET GROUPS 



Audience ; 

Useful with Congress and Educational Publics to show how actual 
progress is being made with specific populations (i.e., degree 
to which needs of East Coast Migrant Stream children are being 
met) and to solve specific problems (re: Identification of 
Methods used to raise reading scores of children in all prog- 
rams), Useful for representatives of minority group popula- 
tions to provide measures of progress and with specific educa- 
tional public and general public (i.e., Remedial Reading 
Teachers) to show the relative value or effect of various 
tec'uniques or approaches. 

C ontent : 

This series of reports would focus on the specific target 
populations and educational goals mandated by all the various 
legislative titles. Some would concern themselves with pupil 
progress; i.e. changes in attitude toward learning, improvement 
in reading scores, better attendance records, etc. Evidence 
of such changes would be cited, within and across titles, and 
specific correlative factors would be cited where possible 
(i.e., correlation between breakfast and attendance; reading 
material type and reading score gains, etc.) 

Such reports would also attempt to determine the. combined 
effect of multiple programs on populations cf children with a 
common need. 

Next steps : 

Prepare list of possible titles using the titles and output 
chart; work up sample report; develop priorities among 
proposed reports; Review concept within OE, 
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EXHIBIT 11-g 

CONTENTS OF REPORT BY TARGET GROUPS (EXAMPLE) 



Report of Services to Disadvantaged 

A. ESEA Services 

ESEA I General Aid Program 
ESEA I Granted Project 
ESEA ill Supplementary Services 
ESEA VII Bilingual Disadvantaged 
ESEA VIII Disadvantaged Dropout 

B. 0E0 Supported Disadvantaged Follow Through 

C. Vocational Services: 

Vocational Education Research Projects 
Smith/Hughes Grants Projects 
George/Barden Grants Projects 
Vocational Education Act 1 63 Support 
Work Study Program Support 
Residential Vocational Education Aid 

D. Summary: Combined impact of Services to Disadvantaged 
Children 
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from the JCES, These begin with r specification for the Special Purpose 
Data Sheet report type, Exhibit 11a which would consist of raw or merely 
summarized data drawn directly from the JCES data base, to be disseminated 
directly to the requestor (or routine recipient, If pre-deflned) at which 
ever level required. This type of data sheet report could be generated 
directly by the JCES when data become available or on request If a non- 
routine request for special data. Both data contents and formats will 
have to be developed for what will probably be a large number of such 
data sheet reports. These can only be defined by the specific users, 
and this development will require a large amount of effort. 

Other preliminary specifications presented below are for reports 
defined as: 

1. Grants Management Information Report (&th. lib) 

2. Program Administration Report (Exh, He) 

3. Management Procedures Dissemination Report (Exh. Ud) 

4. Evaluation Reportsi Title - By - Title (Exh. He) 

5. Evaluation Reportsi By Target Groups (Exh. Ilf) 

An example of the possible content of a report of this last type Is 
shown as Exhibit 11-g. This Is an extremely preliminary set of 
specifications and will be revised and extended In later studies of the 
specific report needs. 

The process of development of more detailed report specifications is 
actually the first step In the development of the Master Data Analysis 
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Plan and is discussed In some detail In the next section. 

Master Data Analysis Plan: Development Requirements 

The heart of the JCES's capability to produce the various reports 
of statistical, fiscal and inter-related data from the various Instruments 
of the system will be the detailed data analysis plan to be developed. Only 
through efficient planning and detailed specification of data manipulation 
and data relation requirements can the plan and the system be efficient 
In rapid and accurate production of data reports of any kind from the JCES 
data sets. This section dost not develop the Master Data Analysis Plann 
required for this purposes} it simply defines some of the required char- 
acteristics of such a comprehensive plan and outlines the developmental 
steps necessary to Its design and completion. 

Data Analysis Plan Requirements. 

Any efficient data analysis plan must perform several specifiable 
operations or tasks to prepare to yield the desired products. It must: 

1, Identify the output information requirements of each de- 
fined analysis process or requirements - that Is, specify the information 
products required for each analytic step. 

2. Identify and Isolate each data element (item response) 

Involved In the Information requirement for the output • that is, locate 
and store for ready retrieval each element required to enter into the 
product . 
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3. Identify and specify the combinatorial or computational 
steps necessary to turn the data elements into output information elements* 

4. Define the exact processes for manipulation of the combined 
data and preparation of the output information in the desired formats, 
etc., for the initial requirements. 

The JCES Master Data Analysis Plan (MDAP) must accomplish these same 
steps, but instead of defining these for a single set of data elements 
(item responses), it must consider interrelationships of the data sets 
derived from the separate instruments, as veil. This will require a 
more extensive job of data identification (Step 2) than might be requred 
for a single instrument analysis plan. However, this io largely an 
enlargement of the normal process of Step 2 and is entirely dependent upon 
the successful accomplishment of Step l - the successful identification 
of the exact information required for the product, When the detailed 
products have been specified, the analyses to determine data element 
identification and manipulation requirements (Steps 2 and 3) become 
relatively direct processes; although the selection of specific analytic 
techniques and the determination of data utilisation can both become 
difficult, time consuming activities. This is also true for the manipu- 
lation processes for producing a report from the combined data elements. 

Development of MDAP for JCES 

Before detailed discussion of the development cf the MDAP an overall 
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view of the expected relationship between the MDAP and the required JCES 
data processing system should be examined. Exhibit 12 presents such a 
view, showing the MDAP (and its developmental steps) in direct relation- 
ship to a postulated JCES information system. The Exhibit shows the basic 
Data Base Development occurring at the Left of the diagram, with data 
input from the JCES instruments being converted to punch cards and merged 
into tape files which would then enter directly into the formatted Data 
Base. Some collected data, such as narrative reports not easily processed 
into the main data base, would be shunted to an ancillary data base for 
later availability to evaluation or other users, either within or 
external to USOE. The main formatted data base would become the 
reservoir of all numerical or categorical data inputs from each major 
JCES Instrument. This reservoir would then form the basis for all report 
generation activities through the System Interface shown to the right 
of the Computerized Data Processing section of the Exhibit. 

Report generation (other than raw data dumps) v jld always Involve 
the application of the Master Data Analysis Plan, shown at the upper 
part of the Bchibit. The MDAP will serve the major function of director, 
or data analysis executive, for all data manipulation and reports 
generation by the system. Whether the HEAP becomes computerized or not 
will depend largely upon the data processing system design approach. It 
could be developed as either an on-line or off-line portion of the 
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computerized system. In either case, its influence roust be the same* only 
the personnel required and the actual operations would be different (that 
is, both programmers and research analysts would be required in the com- 
puterized version whereas only the research analysts might be needed for 
off-line operation.) This influence of the MDAP is indicated in the 
Exhibit at the interface between the MDAP Development block (top) and the 
Computerized Data Processing block. This indicates that the Analysis Plan 
will Either directly or three programmers) direct the input-output (I/O) 
operations of the computer as well as the specific statistical computations 
to be performed on data elements, also specified by the MDAP. It should 
be noted that this data processing system conceptualization is based on 
preliminary data and primarily those related to instrument processing, 
analysis requirements, and reports requirements. 

The performance of the four steps listed earlier for MDAP development 
are indicated by the five bubbles at the top of the upper block of Exhibit 
12. These are shown *8! the interactive determination of information 
requirements for report specification development, with the interaction 
being between the report users and the analysts (Step 1); the development 
of data element specifications to meet the information needs of the re- 
port specif lcatlon(s) (Step 2); the specification of the mathematic and 
statistical computat lonal/analystlc procedures necessary to convert the 
data elements to the required informational forms (Step 3); and, the 
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design snd specification of report formats to meet the user &' specifications 
(Step 4), 

Performance of these four steps through interactive consultation 
with the report users will accomplish the major development of the MDAP, 

The actual Iterative process for performance of Step 1 will be discussed 
in detail to illustrate the approach. Since the data outputs of. each 
analytic process will be the information requirements for a given data 
report, the iterative process illustrated in Exhibit 13 will define the 
products of the analysis as well as the detailed information requirements 
for each report* The Step 1 process starts with report specifications 
such as those presented in preliminary form in the last section and 
carries out a series of iterative steps to obtain detailed ppecif ications . 

As indicated, the first step is to review the preliminary formats and 
proposed information contents with knowledgeable report users in USOE/BESE 
and with some of the State representatives of the Belmont States. This 
is intended to determine the exact information requirements for the 
various reports, the desired formats, and the frequency of reporting 
to be required for each type. When the report user has defined these exact 
requirements, the analyst must consult the JCES instruments and the data 
analysis plans to determine whether the report requirements can be met. If 
so, the analyst would then prepare a series of draft report formats and 
consult with the user until the final draft were approved. This final 
draft would then be made a part of the JCES data processing specifications 
and could be programmed for production runs as required. If the data 
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wer6 not found to be available or producible within the currently planned 
data collection and analysis, the report user would be consulted to deter- 
mine whether the specifications could be modified; if so, a modified 
report would be drafted and eventually approved by the user. If the 
report specifications could not be modified, other data source would be 
checked for the requirements of the user; if found, these data specifica- 
tion generated as above. If no other data source were found to meet the 
requirement, the JOES instruments would be revised (after the usual 
requirements of forms clearances, etc.) and the report specification 
and final formats would be derived through iterative consultation with 
the user, as above. 

For simple data sheet reports, this process i3 seen to be fairly 
simple, given that the needed data are being collected in existing JCES 
instruments; but for more complex reporting requirements, involving detail- 
ed analyses or interrelations of several kinds of data, the report specifi- 
cation development may be rather difficult and time consuming. For 
management data, the modifications to instruments might be considerable 
efforts, also. Or they might force more frequent collection of some 
data types, with more applications of the relevant instrument to the pop- 
ulation of concern. 

In any case, this iterative process will allow final detailed 
specifications for all required reports to be developed. These specifica- 
tions will actually contribute as weel to the final development of both the 
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MDAP and the report generation system* 

This iterative process will partially accomplish the Step 2 effort 
also, since the examination of the individual instruments' item contents 
and responses will generally define the data elements entering into the 
product. For a simple data tabulation or cross tabulation, this iterative 
process will entirely define the data elements required. For more complex 
analyses, the report information products will imply all the elements 
involved and the examination of the instruments will lead to exact 
specification of these through an iterative process similar to the above. 
Report specifications will also be the basis for specification of the 
combinatorial /computational manipulation of date in Step 3, and the for- 
mat specifications will imply the manipulative requirements of Step 4. 

Each of these steps must be laid out in detail prior to further develop- 
ment of the MDAP, However, the basis approaches described here will be 
followed throughout further work on the MDAP. A major assist to this 
process will be derived through incorporation of some current detailed 
analysis plans for the individual instruments (e.g., Dyer & Fortune Plans 
for PCI-1969 version.) 

Some Preliminary Data Processing Consideration 

The preliminary data processing system depicted in Exhibit 12 above, 
is an extremely preliminary concept. However, it is based on current 
knowledge of the JCES instruments, implementation plans, multiple user 
requirements, etc., and therefore has some relations to reality; how much 
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remains to be seen. This development and consideration of the data 
processing system's potential operations have led to additional considera- 
tion of a set of possible difficulties or at least decision points which 
must be passed prior to actual system development and implementation. 

First, it appears obvious that the reports generation required of 
this system dictates a rather large data storage processing, and analysis 
development, This is indicated by the extent of the data accruablr; within 
the system in a single year, alone. However, the potential size of the 
data bank to accrue over years of instrument application, (as is required 
for sequential analyses of Program activities), will be much more exten- 
sive. When this requirement is considered together with the time 
limitations which are currently imposed for deporting on evaluation of 
Programs to Congress, this implies a highly sophisticated and highly 
integrated lata processing development. This is especially true if 
frequent requests are anticipated processing development. This is espe- 
cially true if frequent requests are anticipated to be put to the data 
bank, and if these cannot be predefined so as to ease the burdens of 
"data spinning 11 necessary to obtain selected data bits within a large 
data base. This latter point refers mostly to the "inquiry 1 '' capability, 
discussed under reports. Such a development would allow the research and 
evaluation personnel, as well as all manager^ to enter the data base with 
specific questions which were not predefined as necessary outputs and to 
obtain the data requested without weeks of delay due to special programming 
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requirements. While this and similar types of data availability are 
highly desirable, there is a real question of developmental time and costs 
as to whether these techniques should be incorporated into the JCES develop- 
ment . 

Another major consideration related to this data system development 
was obliquely referred to above - the frequency of reports generation 
to meet the defined requests. Since most instruments will be applied 
only annually (or less frequently), this may be a determining factor. 

But if the SEA-MES is to operate to provide management information outputs 
to the SEA's and LEA's which are current and therefore meaningful, the 
frequency of such report generation probably should be quarterly, and 
perhaps even more frequently. Management information gets old and 
“cold** rapidly, and such information may be of very little value a year 
after the facts. The volume of such outputs must be considered also, since 
if data sheets are to be provided to 20,000 LEA's on a quarterly basis, 
this is rather large data output, and should be processed differently 
than an output of a few hundred printed sheets once or twice a year would 
be. 

A third consideration is the actual degree of "built-in 1 ' stacistical 
sophistication which is desired. A highly sophisticated statistical 
analysis program can be easily handled within the machine capability of the 
current US0E/0MI IBM System/360 machine. But the combination of sophisticated 
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statistical capabilities together with large data bank requirements poses 
somewhat different and more extensive (and expensive) system developments. 

Unfortunately, there are no answers currently available for the questions 
raised in these considerations. They must be answered through further 
analysis of the detailed data requirements and of the desired of the 
USOE and the Joint Task Force with respect to sophistication and rapidity 
and frequency of reporting. They are raised here only because they must 
be answered, along with many other questions, prior to final specific 
data system definition and specification. 
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JCES PROPOSED IMPLEMENTATION 

The JCES is to be implemented over the next several years with 
individual instruments already in effect and others to be developed 
and installed within the next school years. This section discusses 
the proposed implementation plans and the requirements for the 
organization and manpower required to accomplish this. It also dis- 
cusses the proposed development of research/sampling designs conducted to 
this point. 

Scheduled Implementation 

The planned implementation of the various components of the JCES 
is summarized in Table 1 and discussed below. In the Fall of 1969 the 
CPIR was administered to collect Fiscal r 69 data. This administration was 
made in all 50 states. In the Spring of 1970 the Elementary Pupil Centered 
Instruments will be administered to collect data on the , 69- , 70 school 
year to grades 2, 4 and 6 in all states. It is also possible (low 
probability) that the elementary level Common Status Measures will be 
administered in grade four only. 

In the Fall of 1970, uhe elementary/secondary Project Descriptor 
Instrument will administered, collecting data on the school year of 
1970-71, and this administration will be only to the Belmont states. 

Also at this time, the CPIR will be re-administered to collect data for 
fiscal year '70 in all 50 states, and the Common Status Measures are 
scheduled for administration in the Belmont states for grades 4 to 11 
(again perhaps doubtful). Finally, the Project Reference File question- 
naire will be administered in the Belmont states, if it is possible to 
add these two questions to the National Center for Educational Statistics 
standard questionnaire. If not, it may be necessary to administer these 
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two questions separately. 

In the Spring of 1971 the Pupil Centered Instruments are tenta- 
tively scheduled for administration although it may be decided that 
these instruments will be administered biennially instead of annually. 
If administered they will be given to grades two, four, and six in all 
50 states covering the 70-71 school year. Common Status Measures 
(elementary and secondary) are also scheduled for administration in the 
Belmont states. For the Fall of 1971 the schedule for administration 
is the same as it is for the Fall of *70, and in the Spring of 1 7 2 the 
schedule is the same as the Spring of f 71 with the addition of the 
secondary level Pupil Centered Instruments, grades 9 and 11, in the 
Belmont states only. 
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TA^LE 1 

IMPLEMENTATION SCHEDULE FOR JCES 



Time 


Instrument 


Sampled Group 






Fall 1969: 


CPIR 


50 States 






Spring 1970: 


Elementary Pupil Centered 
(FY 69-70) 


Grades 2,4,7; 50 


States 




Common Status Measure 


Belmont - Grade 4* 




Fall 1970: 


Project Descriptor , 
(FY 70-71) 


Belmont 








CPIR (FY 70) 


50 States 








Coiranon Status 


Belmont - Grades 


4 


& 11 




Project Reference File 


Belmont*** 






Spring 1971: 


Common Status 


Belmont - Grades 


4 


& 11 




Elementary Pupil Centered** 
(FY 70-71) 


50 States - Grad' 


i S 


2,4,6 


Fall 1971: 


Project Descriptor 
(FY 71-72) 


Belmont 








CPIR (FY 71) 


50 States 








Common Status 


Belmont - Grades 


4 


& 11 




Project Reference File 


Belmont*** 






Spring 1972: 


Elementary Pupil Centered 


50 States - Grades 


2,4,6 




Common Status 


Belmont - Grades 


4 


& 11 




Secondary Pupil Centered 


Belmont - Grades 


9 


& 11 



* There is still some question about the administration of these, 

** A decision may be made to make the Pupil Centered Instrument 

biennial: this would then be postponed to 1972, 

*** Some parts of this file are derived from the NCES Universe file 
covering all 50 states. 
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Belmont Working Organization and Staff 

The Belmont JCES working organization is shown in Table 2 . Karl 
T. Hereford and Michael Russo, both of U.S.O.E., are co-directors of 
the Joint Task Force, members of which are shown in Table 3 . The 
design team is directed by Daniel Antonoplos of O.E. Its membership 
is shown in Table 4 . Table 5 shows the Support Team. The Coordinating 
Committee is headed unofficially by Mr. Charles Nix of Texas. Under 
this are three working committees each chaired by a representative 
from one of the States and composed of both federal and state personnel. 

The Program Information Committee with Mr. Jess Elliot of Georgia 
as chairman is concerned with the consolidation and improvement of the 
system for reporting the statistical information required by the multi- 
tudinous pieces of legislation covering the activities of the O.E. Bureaus. 
The Consolidated Program Information Report (CPIR) was developed to 
make it possible to report on one set of forms information concerning 
the flow of dollars to services and to target groups from several 
legislative funding sources. The CPIR will eventually allow for the 
collection of information at the district level which will show the 
impact of the flow of dollars by source and the flow of services to the 
various target populations. The target populations will then be the 
primary unit of analysis for the interpretation of the data. 
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TABLE 3 



JOINT TASK FORCE 

K. Hereford, Co-Director 
M. Russo, Co-Director 

E. Crawford 

F. Harrison 

L, laMoure 

A. Lichtenberger 

G. Mayeske 

L, HcGuinness 

H. Pitts 
D, Lose 

K. Wallmau 
K. Willis 
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TABLE 4 
DESIGN TEAM 



Daniel Antonoplos, Chief Evaluation Design 



Consultant Staff: 

Henry Dyer, Educational Testing Service 

Gene V. Glass, Laboratory of Education Research 

Bob Ha». jnd, Ohio State University 

Robert Heath, Stanford University 

GerelJ C. Helmstadter, Arizona State University 

Phillip Kearney, Michigan 

William Maddov, Stanford University 

David Orr, Scientific Educational Systems 

Robert Stake, CIRCLE 

Dan Stufflebeam, Ohio State University 

Wlllavene Wolf, Ohio State University 



Office of Education Staff: 

N. Bradley 
C. Hammer 
I, Llpkovltt 
S . Smith 
L. Tonic 
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I 



J 

Dorothy Gilford 
Harry PiccarielJo, 
Robert Kane, 

Beth Drake Roy, 
Richard Carlson, 
Leon Schwartz, 



TABLE 5 
SUPPORT TEAM 

National Center for Educational Statistics 

O.B. Office of Program, Planning, and Evaluation 

O.E. Office of Management Information 

Bureau of Budget 

Bureau of Budget 

O.E, Office of Administration 
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The School and Pupil Centered Evaluation Instruments Committee, 
chaired by Dr. Phillip Kearney of Michigan, Is concerned with evaluation 
data reflecting on the outcomes resulting from the flow of dollars to 
schools in the form of services and activities to pupils. Thus, a 
set of instruments was needed to obtain information which would describe 
the activitiea and services provided in the achools, the context in which 
these various programs were offered* and the outcomes for varioua types 
of individuals who participated in the services and activities. This 
effort is an attempt to relate the various kinds of students participating 
in different programs to the outcomes which occur. Rather then building 
on the national surveys which had been undertaken in past years to eval- 
uate four Federally funded programs, a new inatrument is in the process 
of development which will provide a comprehensive evaluation of ten 
programs: Titles I, II, III, V, VII and VIII of ESEA; Titles III and 

V-A of NDEA; Vocational Amendments of 1968; and Title IV of the Civil 
Rights Act , including these four. 

The third working committee is the Evaluation Training and Local 
Evaluation Conraittee, chaired by Mr. Charles Nix of Texas. The conmittee 
haa the responsibility for exploring long range evaluation needs at the 
state and local levels. In dealing with this responsibility, the members 
of the group agreed that assistance in providing orientation and training 
in evaluation for local staff personnel was necessary. Therefore, part 
of their effort was the design of several one-week institutes intended 
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to acquaint state and local personnel with evaluation tools and techniques* 
In addition, the committee sought to develop a Guide for use by individuals 
responsible for local evaluation and report preparation. 

JCES Manpower Requirements 

Table 6 shows the O.E. manpower requirements in manroonths for 
the various tasks required for development of the JCES instruments. The 
figures on the table are essentially a summary of the ten individual Gantt 
Charts prepared by U.S.O.E. and reflect full time Office of Education 
employee requirements only. They do not take into account any contractor 
personnel. The total requirement for the period from FY70 through FY73 
is 2489.4 manmonths (357.5 for FY70; 678.7 for FY71; 682.2 for FY72; 
and 771.0 for FY73.) The JCES instruments included in the table are: 

l* Consolidated Program Information Report 

2. Pupil Centered Instrument (Elementary) 

3* Project Descriptor Instrument 

4. Common Status Measures 

5. Program Reference File 

6. Pupil Centered Instrument (Secondary) 

7. State ’Management Report 

8. Consolidated State Application 

9. LEA Consolidated Application 

10. Anchor Test 
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TABLE 6 

HAKPOWER REQUIREMENTS (HANMONTHS) FOR JCES 



777ff 



TASKi DESCRIPTIONS 


3 


A 


S 


0 


N 


D J 


F 


M 


A 


M 


J 


M^AR 


A. Develop 6 Install 


























Instrument 

1. Review 6 Evaluate 
present information 
2* Draft Instrument 


50 


SO 


10 






110 


05 


20 






6D 


36.5 


Content 




25 


35 


20 






90 


45 


20 






23.5 


3. Review Content-OE 






20 


35 


20 05 




45 


30 


20 




17.5 


4. Review Content-SEA 






05 


30 


20 OS 10 




15 


20 


40 




14.5 


.5. Draft Instrument 






18 


35 


15 


2D 15 


05 




35 


25 


ID 


17.8 


6. Approval of Draft 

7. Form Clearance/Field 


20 






C5 


15 


15 20 


05 


05 




65 


45 


19.5 


Test 




05 






30 


. 


30 


15 


15 


20 


40 


15.5 


8. Field Test 




15 


15 




40 


35 05 




10 


10 


05 


20 


15.5 


9. Revise Draft 
9A. Local Target Fop* 






05 


03 


15 


25 05 






Q5 


10 




6.8 


Census Procedure 


























9B. Develop Attribution 


























Manual 




10 


20 . 


10 


05 


10 05 


05 


05 








6.0 


9C« Stratlf icatlon models 


























fqr Instrumentation 
Snmple Designs 


























* 10. Develop Analysis Flan 


























10A. Prepare Table Shells 


05 






05 


20 


20 32 










3D 


11.2 


10B. Roview-OE 


05 


05 






05 


30 












A. 5 


IOC* Revise Table Shells 








03 


05 


20 


15 










4.3 


11. Sample Selection 


40 


20 


80 


ao 


80 


35 fO 


30 










42.5 


12* Final Forms Clearance 
13. Determine External 




10 


20 


10 




20 10 


60 


10 








14.0 


Flow 


03 


05 








05 10 


11 


05 


03 






5.3 


14. Final Instrument 


























Revision 6 Approval 








05 


LO 


15 03 


15 


20 








6.8 


14A. LMR Instrumentation 
148. SMR Instrumentation 
14C* Secondary Instrumen- 


























tation 


























15. Print Instrument 








02 




07 


LO 


LO 


LO 






3.9 


16. Kail Out Date 








03 




03 


02 


05 


06 






1.9 


16A. Validation Studies 
J7. Date due inOS_ 












05 10 


10 




05 


10 


10 


5.0 


B. Develop Requirements 
Specifications 

C. Automatic Data Pro* 

, ceasing 






13 


15 




* 


30 


SO 


35 


15 


30 


19.0 


1. DADF 








10 


2D 


20 10 




20 


30 


60 


60 


23.0 


2. Bureaus 












25 25 










40 


9.0 


D. Final Report 
R* Redeveloping and 












10 20 


30 


20 


10 


10 




10.0 


Recurring Costs 
1 . Instruments Re- 
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TABLE 6 COin*. 

MANPOWER R EQUIREMENTS (MAKMONTHS) FOR JOES : 

FY72 TOTAL 

TASK DESCRIPTIONS JASONDJFMAMJ FOR YEAR 
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TABLE 6 CONT. 

MANPOWER REQUIREMENTS (HANMONTHS) FOR JCES 
FY73 
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The following is a description of the tasks as listed in the left 
hand column of the table: 

Instrument Task Descriptions 
A* Develop and Install Instrument 

1. Review and evaluate present Information - This task encompasses 
the collection and review of forms, informational interviews 
with management, review of plans, etc. 

2. Draft consolidated instrument(s) content - Based on the 
information acquired in the previous task, construct a list 
of proposed data elements. 

3. Review proposed instrument content with OE management - This 
task will require review with OB, Bureau, division and program 
level management. 

4. Review proposed instrument content with SEA representatives - 

the proposed instrument content will be reviewed initially by the 
States represented in the joint SEA/OB Task Force on Evaluation. 
Comments will be solicited on the reportablllty of the data re- 
quested and relevance to SEA needs. Regional meetings will be 
conducted after the initial Task Force review to involve the 
remaining states. 

5. Draft instrument format * The information acquired in the three 
previous tasks will be codified and evaluated and a draft lnstru* 
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ment will be prepared. 

6. Obtain approval of draft instrument - This task will repeat the 
review channels utilized In steps 3 and 4 above in order to 
obtain preliminary approval. Any necessary revisions will be 
made during this task. 

7. Forms clearance for field test - Obtain BO? clearance for field 
test of draft Instrument. 

8. Field test - The Instrument will be reviewed with intended 
respondents to determine feasibility of Implementation and 
use. 

9. Revise draft - Based on experience gained in field test, revise 
Instrument content and/or format and prepare Instrument In ADP 
format. 

10. Develop analysis plan - Draft table shells 

a. Draft table shells • TMs task will relate the Input data 
elements to the previously determined management Information 
needs In the form of output table shells. 

b. Review with OE management - The proposed tables will be 
reviewed and cleared with all levels of management In OB. 

c. Revise table shells » Table shells will be revised, If 
necessary, to reflect management comments received In above 
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11. Sample Selection - Specifications for the sample will be for- 
warded to NCES or a contractor and a sample will be drawn 

0 

based on the purposes the instrument is designed to serve. 

12. Obtain final forms clearaace - This task includes tie preparation 
of a package containing the instrument, narratives describing 
data utilization, and the sample plan and clearance through 

OE forms cleamace channels and the Bureau of the Budget. 

I 

13. Determine external flow - The procedure flow and responsible 
channels for mailing and return to OE of the instruraent(s) will 
be determined. 

14. Final instrument revision and approval - Revise instrument, 

as required, by fora clearance authorities. Prepare requisition 
for printing and mailing and obtain administrative clearance. 

15. Print instrument - This task encompasses the tine line for 
monitoring and actual printing of the lnstncent through the 
Government Printing Office. 

16. Moil out - Accomplishment of this milestone will in all cases 
be keyed to the success of the total implementation to the 
DHEV Printing Office. 

17. Due date to OE - This milestone indicates the date specified 
on the instrument for return to OE or a designated contractor. 

* Normally this is also the date follow-up of late respondents 




\ 



will begin. 



125 



B, Develop requirements specification - This specification will include 
cs a minimum; the input instrument format and data definitions; the 
output formats and data relation to the input, the internal flow 
and control procedure, the specific time requirements, and the 
interface requirements with previously developed instruments sub- 
systems . 

C. Automatic Data Processing (ADP) 

1, Automatic Data Processing Division 

A, Develop ADP system specifications 

a) system flow chart(including, interface with other 
sub-systems), 

b) system computer configuration 

c) manpower, time, and cost analysts 

B, Prepare package for contracting, if required. 

C, Develop sub-system(s) 

a) detail design 

b) programming 

c) testing 

D, Develop system 

a) integrate sub-sys tem(s) 

b) system testing 

E, Review system results with management. 
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F. Revise system as required* 

G. Complete system documentation for production control, 

H. Implement system 

a) capture data (keypuch, etc,) 

b) training 

c) production 
2, Bureaus 

A, Editing and eiror correction 

B, Follow-up on respondents 

C, Review of sample system outputs with ADP 

D, Training in use of system's inputs /outputs 

D , Prepare Congressional Reports and Program Publications-Most instru- 
ments have as their primary end product a report to Congress, This 
task involves the selection and preparation of significant tables 
for inclusion and a narrative interpretation of the data, The 
3 teps in preparation involve program officers, information 
specialists and Division, Bureau and OE management levels* 

E, Instrument Redevelopment and Recurring Costs 

1. The instrument must be reviewed each year to take into account 
changing legislative and program information requirements and 
experience gained In implementation. The steps involved in 
initial instrument development must be retraced on an annual 
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basis to accomplish the redevelopment task. For example, 
program officers, Bureau management and OE management must be 
reinterviewed to insure instruments are meeting their needs, 

2, A constant data processing effort of system maintenance and re- 
development will be required in order to maintain systems 
that are responsive to evpr changing r.nd increasing require- 
ments, Special output requirement, usually "one-time" in 
frequency, are also included in this task, 

Contractual Requirements 

In addition to the O.E, staff members that will be required, outside 
organizations under contract will be used to assist in the development 
and implementation of tl:e system's data collection instruments. 

For the Consolidated Program Information Report (CPIR), contractors 
will be used during FY70 to revise the draft instrument, develop the 
attribution manual, assist in the development of the data analysis plan, 
print and disseminate the Instrument, conduct validation studies, assist 
In the data processing and prepare the finul report. During FY71, 
contractors will continue with the development of the attribution manual 
for the CPIR, print and disseminate the Instrument, conduct validation 
studies, assist with the data processing, and prepare the final report. 
During FY 72 and FY73, contractors will be required to conduct validation 
studies on the data collected by the CPIR, assist with data processing 
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and preparation of the final report, and help with any instrument redevel- 
opment. 

For the Pupil Centered Instrument (Elementary), contractors will be 
used in FY 70 to prepare the instrument content, assist with the data 
analysis, print and disseminate ;he instrument, conduct validation studies, 
assist in the data processing, and prepare the final report. During 
FY71, 72, and 73, contractors will assist with the data processing, 
prepare the final reports, and help with any instrument redevelopment. 

For the Project Descriptor Instrument, contractors will be required 
in FY70 to prepare the instrument content and assist with any required 
content revisions. During FY71, contractors will be needed to print and 
disseminate the instrument and assist with the data processing. During 
both FY72 and FY73, contractors will be used for any instrument redevelop- 
ment tasks and for data processing. 

For the Common Status Measures, contractors will be used during FY70 
to draft the instrument content, print and disseminate the instrument, and 
assist with the data processing. During- FY71, 72, and 73, contractors will 
be required for assistance with the data processing and for any instrument 
redevelopment(assuming the CSM are retained as a part of the SCES). 

During FY70, contractors will be used to assist with the sample 
design for the Program Reference File. During FY71, contractors will be 
needed to print and disseminate this instrument, assist with the data 
processing and prepare the final report. Contractors will be used 
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during FY72 and 73 to assist with the data processing, prepare the final 
report, and help with any instrument redevelopment. 

For the Pupil Centered Instrument s (&e condary) contractors will be re- 

* quired during FY71 to review and evaluate pi ins and information for the 

development of the instrument. During FY72, contractors will assist 
% with the data analysis plans, print and disseminate the instrument, :on- 

duct validation studies, assist with the data processing, and help with 
any instrument redevelopment. During FY73, contractors will he required 
for the preparation of the final report, and for assistance with any 
instrument redevelopment and data processing. 

For the State Management Report, contractors vi^l be used during 
FY70 to review and evaluate plans and information for the development 
of the instrument. During FY72, contractors will be used to print and 
disseminate the instrument and to help with any instrument redevelopment. 
During FY73, contractors will be needed for the preparation of the final 
report, for assistance with data processing, and for help with instru- 
ment redevelopment. 

For the Consolidated State Application, contractors will be required 

* during FY70 to review and evaluate plans and information for the develop- 
ment of the instrument. During FY71, contractors will help with the final in- 
strument revision, print and disseminate the instrument, and assist with 

the data processing and any instrument redevelopment. During FY72 and 73, 
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contractors will be needed to assist with data processing and Instrument 
redevelopment. 

For the LEA Consolidated Application, contractors will be required 
during FY71 to identify the local target populations, assist with the 
final instrument revision, print and disseminate the instrument, and assist 
with the data proc osing. During FY/2 and 73, contractors will assist 
with data processing and any instrument redevelopment. 

Finally, contractors will be needed to conduct test studies during 
FY70, 71 and 72, to develop score equating for reading te3ts, math tests, 
and other subjects. 
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Political and Other Considerations in Implementation 

This Motion sterna from discussions with USOE. Task Force members and 
from analytio review of the evaluation system problems taken in relation 
to previous experience. Some of these problems concern somo of the poli- 
tical and practical considerations which must enter into development of 
an enterprise of this nature and scope. The section attempts to sum- 
marize the important considerations in this area but does not essay 
solutions to these problems. 

f/Otae Political Considerations 

Many considerations are involved in the planning and development of 
an evaluation system and not the least of these is the relationships which 
exist between the evaluators (USOE) and those who may tend to feel they 
are being evaluated (The States and LEAS). Such relationships are extremely 
important to the success of this system, since all input data comes from these 
sources and their cooperative contributions are essential to success of 
the system. Under current circumstances, the USOE must negotiate 
separately with each of the fifty States in order to gain cooperation at 
that level, and further there must be a separate transaction with each 
school system or district which is to be contacted; still further, in 
some cases it is necessary to negotiate the cooperation directly with the 
schools within a district. Other problems may arise at times in the 
attempted application of a survey instrument, e.g. , the sometimes difficult 
negotiations with teachers' unions and similar special groups. 

At the same time, the current relationships of USOEjand the Federal 
Government in general, with the States and local agencies periodically 
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become somewhat strained d-e to the heavy workloads imposed by the present 
reporting requirements, referred to earlier* States, districts, schools 
and teachers all tend to resent the imposition of any additional work- 
load and in many cases this resentment builds to concrete insistence to 
performance of the requested tasks. Therefore, insofar as possible, the 
developing JCES should be based on the principle of non- increased workloads 
cn teachers, schools, districts or State agencies, and it should be 
designed to reduce these loads wherever possible, without sacrifice of 
data quality and sufficiency* 
iiome Practical Limitation on USQE 

A major factor affecting the conduct and reporting of any evaluative 
effort is the basic requirement for an annual report to Congress of these 
efforts - due 31 January of each year. This requirement, with the due date 
in the middle of the school year, works to force the USOE to conduct 
either M old evaluations” f^data from last year being evaluated) or ‘’quick 
and dirty” evaluations of the current year’s data. Evaluations based on 
surveys which must be mailed to school, or districts, and then must be 
returned, analyze!, interpreted and reported, are simply not easily done 
and, when the whole process is limited to a few months of time, they are 
unlikely to be either comprehensive or reliable. Thus any survey 
initiateu in the Fall is threatened with inadequacy because of the stringent 
time period for completion. The only possible answers to this problem 
seems to be either to remove the limitation of time for data collection 
and reporting - through later reporting dates or perhaps biennial 
evaluations - or to develop so comprehensive and coordinated an evaluation 
process that the whole job of data collection, analysis, and interpretation 
runs quickly and smoothly from start to finish, with no hitches (and 
probably with no possibility for asking or answering any unique questions 
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in any application of the survey), Such a completely routinized evaluation 
system would undoubtedly handle most of the evaluation requirements of 
the UbOU and related agencies, but it might well result in a rather etag- 
nent evaluation process, as well, which might partially negate the advantages 
of ths process. 

liven though there are some serious drawbacks to a routine evaluation 
system such as envisioned above, this concept of a coordinated, complexly 
interrelated, relatively routinized system should probably form the he&rv 
of the new JCKo. This should be augmented with an innovative, interrelated 
inquiry system allowing derivation of answers to non-routine questions and 
should also provide a methodology and techniques for updating and modifi- 
cation of the instruments used for data collection from year tc year. 

Such a system will not eliminate certain other internal problem areas 
related to system development and application, and these may still force 
a real, if not legalized, requirement to fall back to a two year system 
of reporting despite the routinized data collection and evaluation capability. 
These internal items are certain regulations and practices related to the 
areas of instrument development or modification, survey conduct and local 
agency cooperation, and report preparation aird clearance for external 
release. For example, in the development or modification of a new survey 
instrument alone, there are multiple internal hurdles to clear: 

1) First, the development of a major instrument requires a 
contractual arrangement for development and standardization. This requires 
an RFP and contractor selection process, which can rarely be accomplished 
in less than 60 days. Contributing to, and probably adding to, this time 
are the three internal levels which must be cleared before funds can be 
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releasods a) USOE/Bureau level} b) Plans and Programming Secretariat (USOE) 
ard c) Bureau of the Budget# These clearances may be quickly or slowly 
obtained and variations are unpredictable# Belay at any level lengthens 
the period prior to contract agreement# After the contractor is aboard, 
the delays necessary for actual development or modification of the 
instrument ensues# 

2) As part of the development and application of any instrument, 
it is necessary to have the National Center for Educational Statistics 
(NCES) review or develop the research/sampling design for the use of the 
instrument# This requires internal coordination and frequently adds to 
the delay in application# 

3) Following completion of the instrument and approval or 
construction of the sampling designs by NCES, the instruments themselves 
must be cleared for application in the field* Again there are multiple 
levels of clearance; USOE clearances include, again, NCES and the PPE 
Secretariat, and the form must be cleared by the Bureau of the Budget as 
well, unless used strictly for research purposes and not to be routinely 
applied# These clearances add again to the period of time necessary to 
develop and implement any survey instrument* 

Report preparation and submission is subject to a similar series of 
examinations and reexaminations which include clearance by; a) NCES for 
statistical appropriateness 5 b) UBOE/PPE for substantive treatment and 
clarity of reporting; and, c) the Press Office must clear the report for 
general acceptability and concordance with other related data reported 
or existent. These delays add to an approximately six to nine month lag 
between the receipt of a contracted or internally prepared report and its 
delivery to Congress or other external users* 
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Evaluation Criterion Problems 

The JCES implementation will encounter all of the usual problems 
associated with the development and implementation of any evaluation 
proceee or cyetem. These certainly include the basic measurement 
problems related to the instruments and their analysis, such as reliability 
and validity | but these are complicated here by the multiple criterion 
nature of the process and products which are to be evaluated* This 
relates to the basic difficulty of adequate measurement for each of the 
program processor under examination, but also extends to the problem of 
attribution of effects to probable causes, since the Congress and the 
Public want to know whAt they have accomplished for their money spent. 
Cause-effect identification is always difficult but will be especially so 
in the situation described in the Delivery System. Kith the Funding 
Subsystem and the Substantive Program Subsystem overlapping to the degree 
indicated, one will seldom be sure which of the various funding programs 
have contributed most (or any) to a particular Substantive Program and 
tho resultant changes occur in pupil behavior. 

Thus, substantive programs may be evaluated by assessing changes 
in the pupils, or the communities, if so directed. But these changes, or 
lack of changes, might not necessarily reflect the values of the Funding 
Programs, since usually a true substantive program results from funding 
under several of the various Titles. Another aspect of this difficulty 
1 b that the Funding Subsystem may define the general area of even the 
intended content of a Substantive Program for application, but the actual 
presentation of the Program may be c^>neiderably different from what was 
intended and approved. Obviously, euch changes may be good or they aay 
be bad, but the fact of changn negates the ues of the products of euch a 
Substantive Program ae valid meaeurea of tha Funding Subsystem'* effective- 
ness. Such Programmatic changes aay be directly indicative tf the quality 
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of local management or directiveness! and might be ueed to assess needs for 
modification of management procedures or in-service training projects) but 
they are not sufficient measures of the total management structure's suc- 
cess of failure nor should they be used as sufficient indicators of a 
F unding Program's total worth. 

( Implementation of Surveye 

Another factor affecting the development and implementation of this 
system is the set of variables related to the management and administra- 
tion of the surveys. Since all surveys are mailed to schools or district?' 
the adequacy and completeness of the data derived are dependent upon the 
conscientiousness and competence of the persons completing the survey 
instruments. The evaluation of the total delivery system, including both 
the Funding and Substantive Program Subsystems, is entirely dependent upon 
the willingness and skills of persons about whom these qualities are 
usually almost unknown. Because of these local survey administration 
problema, US03 takes the position that the best approach, where possible, 
is to establish the responsibility of one person for local completion of 
the inetrumente and their return to the USOE. This precaution is likely 
to reBult in both a higher rate of return of data inetrumente and a higher 
degree of reliability and validity of the data entries. 

» A related point concerns the kind and degree of data "editing” or 

local quality control - of both the data reported and of the pupils and 

* projects on which they are reporting - which may be exercised in the 

process of instrument completion. While instructions always attempt to 
standardise euch processes (and to eliminate the eelectlon of especially 
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favorable data, for example) thle facet of data collection ie essentially 
beyond the control of the USOE. The beet which cwi be done ie to develop 
complete and concise inetruotione for euch collection and to attempt to 
estimate the degree of veraoity through' related reliability and validity 
studies on subsamples of the original groups assessed. An adjunctive 
approach ie poseible through the development and implementation of 
standardized adminietration/evaluation guides (like the Local Evaluation 
Handbook, developed by AIR and intended for use in conjunction with the 
JC£3) or of local in-service training programe on evaluation, survey 
collection requirements, and related topics* 

Iffacte of Biaees 

The above considerations are complicated o till further in the 
evaluation attempt by the various local or individual biaeee to be found 
with reepect to educational objectives, Federal intervention, evaluation 
purposee and many other topics, of which the biases for or against tests 
and testing is not the least difficult to deal with* Although all such 
biaeee will affeot the degree of cooperation and the adeq\acy of the data 
obtained from State and local agencies and in the schools, themselves, 
testing biases, and feare related to testing results, pose a uniformly 
difficult problem* State and local personnel frequently react negatively 
to the concept of (perhape unfavorable) comparisons which they feel may be 
inherent in any evaluation syetem* Such reaotions have led States and 
districts to deny the implementation of instruments in their systems 
before and may do so in the present instance* 
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All of tho above considerations oust influence the development and 
implementation planning for the installation of the Federal/state Joint 
Comprehensive Evaluation System* 
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Kesearch/Samollng Designs: 

Current Developments 

The current Implementation schedule, presented earlier in this 
section, indicates that the CPIft has already been applied to a sample 
of s.nool districts. Similarly, the I9d8 Survey of Compensatory 
4a.' rat inn and the 3 969 Survey were applied to samples of schools in 
tb- r-yring of the respective years. These Surveys are the forerunners 
of the JCES Elementary level Pupil Centered Instruuenta (PCI) in the 
Spring of 1970, again to a sample of schools and a sample of children 
within schools. 

For each of these instrument applications, the National Center for 
Educational Statistics has developed a sampling plan consistent 
with the research/evaluation aims of the instruments and has drawn the 
actual samples. For example, the 1968 Survey was administered at 
four levels * District, School Principal, Teachers (2nd, Ath, and 6th 
grades), and Pupils (completed by the Teachers) * within A65 districts 
across the country. These A65 Districts were determined by a sample 
plan developed by NCES and were drawn from four strata of districts 
(divided on enrollment population). Following selection of Districts 
via stratified random sampling (with the largest site stratum being 
sampled exhaustively), schools were sampled within Districts in the 
ratio of III. A. Vithin the sampled schools, all Principals were sur* 
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veyed, os were all Teachers of the three grades ccncemed. Each Tescher 
reported on pupils at the approximate rate of 1/5 of tnelr rlass 
memberships. This, in brief, is the sampling proceus whlcn was applied. 
And the same sample was used for the 1969 Surrey of Compensatory 
Education, also. 

Analyses of the 1968 Survey data revealed that the sample yielded 
excessively large Coef flclents of Variation on several Important mea- 
sured characteristics, snd that these were especially large within the 
top most strata (which had included Districts with student population 
of 40,000 and above.) These findings led to the design of a larger 
sample of Districts and school* for the application of the FY 1969 
CP1R (applied in Fall, 1969) and this sample will also be used for the 
collection of the PCI dntn in April 1970. This sample and its develop* 
ment process will be discussed below, since this is the most current 
and thus most representative, of the sampllng/research designs developed 
by BESB/NCES for JCES use. 

The 1969 sample derivation followed generally along the lines 
of the 1968 sample. It utilised a stratified random approach to 
sampling first 830 Districts, (from a list of those District of over 
300 student enrollment and receiving State Plan funds from Federal 
Titles); and, second, schools within these Districts, drawn randomly from 
a list of district schools ordered on their "average wlthln-grade site," 
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for 2nd, 4th, and 6th grades. Strata boundaries used In the 1969 sample 
differed from those of the 1968 sample, as did also the number of pupils 
reported on by Teachers. Certain othe^ variations were observed In the 
sample development, bu^ these were the most Important ones. The 
survey sample was augmented for the FY 1969 CPIR administration by 
adding all LEA's receiving direct grant funds, yielding a total CPIR 
sample of approximately 1500 Districts. This augmentation Is not 
anticipated for the 1970 PCI application, and only the 830 selected 
Districts •..•111 be surveyed. 

The following table describes the characteristics of the derived 
sample of Districts: 



Stratum Boundaries 
(Student Enrollment) 


Number of districts 
in Population 


Sampling 

Fraction 


Number of 
districts In 
Sample 


125,000 


m 




13 


1/1 


13 


35,000 


* 


125,000 


95 


1/1.3 


72 


9,000 


• 


35,000 


699 


1/6 


117 


3,000 


» 


9,000 


2272 


1/9 


252 


300 


m 


3.000 


8671 


1/23 


376 








Total* 11750 
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As may be seen the stiata used were five In number, instead of the four 
sampled In the 1968 Survey sample development. This was done in on 
attempt to reduce the extremely large within stratum variance found In 
the upper stratum (40,000 and up) of the 1968 sample. Breaking this 
stratum, consisting of 92 Districts In 1968, approximately two strata 
of 35,000 to 125,000, (with 95 Districts), and 125,000*plus with 13 
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Districts, should have reduced the variance In the topmost stratum 
to a level which would be more commensurate with the other within* 
stratum variances. In fact the boundaries of these strata were deter* 
mined primarily through the process of Iterative comparisons of within 
strata variances with different stratum boundaries until the variances 
becsme as nearly equal as feasible. It should be noted that the variances 
being manipulated for this purpose were the variances of student enroll- 
ments for districts In the strata examined. That Is, when a given 
boundary set was used, the resulting population was examined across 
strats on the size of the variances withln-strata In student enrollment 
per districts within the stratum. The process wss actually performed 
using the Coefficient of Variation for each "experimental" stiaturo, 
but the desired effect Is the sama as were one to use variances. 

Following the derivation of the above listed stratum boundaries, 
the number of districts to be samples In each stratum was determined 
using the Strata variances In the formal computation to determine 
sample site required.* Each derived sample size was then multiplied by 
11 IX to allow for non-response. This resulted In the Individual strata 
sample numbers shown In the above table, with the highest stratum being 
sampled exhaustively again, as Is a common result In such allocation 

* This computation used the standard + 2 slgnsva Interval to produce 
an n large enough to yield Coefficients of Variation of 51 at the 
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.95 confidence level. 
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techniques . 

Thle derived sample (+ those LEAs receiving direct Federal Grant 
Funds) was sufficient for the CPIR administration, since that Instrument 
Is completed at the District level only. However, for the purposes of 
the 1970 PCI survey, this sample of Districts must be subsampled for 

schools to surveyed. The process to be applied here Involves first 

< * 

the determination of how many schools need be sampled In each district 
and then the selection of these randomly from a list of district schools 
ordered on "average-vlthln-grade-slze" for the three grades of interest. 
The first point, how many schools within districts, Is to be determined 
as follows*. 

1. The number of classes per grade for the three grades 
per school will be determined, within each District sampled. 

2. Since the number of pupils to be reported on will be 
four per class per grade, per school, (all classes of those grades 
within a sampled school will be surveyed) the total number of schools 
necessary depends on the total number of pupils desired In the total 
sample. (This number has not yet been set, when It Is, the following 
steps can be performed.) 

3. Given the total number of pupils within grade levels 
desired In the school sample, this number should be allocated among 
Districts In proportion to the number of clssses (teachers) vlthlr, the 
District for each of the three grades of Interest. 
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4. Since one quarter of the allocated number of pupils 
desired from each District will equal the number of classes required 
for the District sample, this number divided by the total number of 
classes in the District (within grade level in each case), will yield 
a ratio which can be used as the sampling ratio for schools within that 
District and which will yield very nearly the desired number of pupils 
in the District and in the total sample of 830 districts. 

The PCI survey in April of 1970 will send the various level forms to 
the sampled Districts, schools and teachers derived according to the 
processes described above. 

Further Sampling Developments Required 

Although the current Instrument applications are Intended to pro* 
vide only a Nationally representative set of data, furture demands of 
the JCBS and the Belmont States needs will make State representative 
samples much more desirable. This is planned for development for the 
FY 1970 application of the CPIR (Fall 1970 collection.) The cample 
development for that application will .nclude approximately 4000 LKA's 
and will be comprised of State samples for the Belmont States at 
least. This development of State*reprecsntatlve samples is likely to 
continue and to be required for most of the National survey applications 
of the various Instruments. Thus is is predictable that the PCI survey 
in 1971 will be performed on a State and Nationally representative 
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sample. 
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The development of additional sampling plans to meet these needs 
Is n current requirement for development to assist the JCES In meeting 
the commitments to the Belmont States. Therefore, the devdopment of 
such State representative samples of districts and schools must be 
accomplished within the next six months to provide for the proposed 
application of the CPIR to auch State samples. The precise approaches 
to these developments have not been defined, but It may be assumed 
that they will be highly similar to those applied to the recent survey 
Implementations. 

Future Samole/Research Design Requirements 

The following discussion and comments represent additional consider* 
atlon of the problems associated with design of State and National 
samples for the Implementation of the various JCES Instruments. They 
relate directly to an extension of the current designs described above. 
Comments Concerning Future Sampling Plans 
Future sample plan developments for the OCRS Implementation 
require some additional consideration of the problems of meaningful 
stratification factors be sampled to the various strsta or cells of 
the derived designs. Some pertinent comments are presented below. 

1, Stratification Foaccor* Revisited. 

In the sampling plans described above, the only stratifies* 
tlon factor used was the student enrollment of the Districts; student 
enrollment was also to be used to order schools within Districts. 
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This Is not a suVstratlficatlon, but rather an ordering process to 
assure that the sequential draw would yield a dispersion of sample 
schools on enrollment size within District. However, It appears that 
stratification on enrollment, was done because It was the major factor 
on which population data were available for computation of the population 
variances (for use In adjusting stratum boundaries, as described pre- 
viously.) When the survey data become available from the 1968 and 1969 
Surveys of Compensatory Education, the CPIR administration, and the 
1970 application of the PCI's, these data could be used to examine the 
effects that shifting the boundaries of enrollment-alee-stratificatlon 
might have on the Coefficients of Variation for estimates of these 
variables of major interest: e.g , - academic achievement scores or, 
from the CPIR, total Federal funding by lltle and Target group. This 
study could be played off In the same manner In which the boundaries of 
enrollment site strata were determined previously. There the shifts In 
boundaries were to produce near- equivalent Coefficient of Variation on 
enrollment estimates across all strata. This study would be directed at 
shifting school data on the variables of Interest back and forth, from 
one stratum to another, *n the sane manner as above and examining the 
changes In the CV't of the major variables of Interest as a function 
of stratum boundary shifts. This study would provide data to support 
or deny the effectiveness of stratification on enrollment site alone for 
obtaining the desired presltlon of estimation with respect to other 
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variables . 

Obviously, the study suggested would require time and effort on 
the part of NCES, BESE, or OMI, and this expense should not be squander- 
ed. To avoid the necessity for such a study for all variables of inter- 
est, a preliminary inveatigation of the relationships between. the vari- 
ables of interest and district enrollment size should be performed. 
Correlation of district mean values for each variable with district 
size would provide an indication of the degree to which these variables 
are related. This relationship would then indicate the suitability of 
using enrollment as a stratifier. If correlations of r ■ .60 or 
upwards were consistently obtained for all variables, then stratification 
on these other variables instead of enrollment size would be unlikely to 
improve the precision of estimation very greatly. But, if some 
or many correlations between enrollment size and other variable means 
were to be on the order of r ■ .20 to .50, one might suspect that 
stratification on enrollment size was failing to contribute significant- 
ly to precision in the estimates for these variables. In that case, 
the study suggested above should be done to determine the degree of 
effect of sample size boundary shifts on the precision of estimation of 
the variables of Interest. This could conceivably result in coming 
up with better bases for stratification (even if still on enrollment 
size, but with the boundaries determined on the basis of the known 
effects of size variations on other variables.) 
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Another question for consideration is the effect of performing 
the same study using school size and school mean values for the other 
variables. The results of this study might differ significantly and 
importantly from those based on district data# If this were to be so, 
then the use of District based samples for deriving variable estimates 
for schools migh be seriously questioned. 

2. Allocation Methods 

This comment concerns the method used to predict and 
allocate the sample size across a strata. For the most part, surveys 
are aimed at the estimation of parameters of only a few variables, 
usually one or two. The majority of the JCES instruments are collecting 
data and estimating parameters on a dozen or more variables. Experience 
has indicated that when this is the case, the optimum allocation model 
should probably be used to predict total sample size required and allocate 
this number across the cells of the strata. Hanson, Hurwitz and Madow 
indicate that the additional power gained through application of this 
allocation model far outweighs the extra effort necessary to deal with 
the different weighting factors involved.* 



* Hanson, Hurwitz, and Madow, Sample Survey Methods and Theoi’y, 
Vol. 1, New York: Wiley and Sons, . p. 227. 
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3. Double Stratification 

The use of two or more stratification variables can greatly 
Increase the efficiency of the design with respect to the precision of 
estimation on these meaningful variables that ere srtially correlated 
with each of the two variables used as stratifiers, Under a double 
stratification paradigm, the use of enrollment size as an initial strati* 
fier might become more utilitarian if the second strati fier were 
something like socio-economic statue, or some other factor presumed highly 
correlat j with the variables of major interest. Such a doubly 
stratified design can use the optimal allocation model to determine 
total sample size necessary and the allocation of this sample over the 
levels of the main stratification factor, and then this assuredly suffi- 
cient size for each stratum could be distributed over the levels of the 
second stratification factor on the basis of an equal probability model. 
Other more complex techniques are also available for optimum allocation 
to all cells of such a matrix and could be applied, 

4, Overlapping Samples 

An important point to remember in the development of samples 
for the implementation of the individual JOES instruments is the need to 
be able to interrelate estimates and data derived from the various 
instrument* whenever this is possible. This implies that, wherever 
possible, the samples used for different instruments in a given year 
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should either be identical or highly overlapping. This will result in 
development of data from various instruments on the same groups of 
schools, districts and pupils. Such a procedure could have three 
advantages: a) This could allow the direct relation of data from one 

survey to that from another if data are actually derived on the same 
group, permitting better interpretability of the data sets; b) Use of the 
overlapping samples for two different' instruments (or for reliability 
or validity studies on one instrument) could allow the extrapolation 
of data from the subsample to the larger sample through use of the 
hyper-geometric sampling distribution for extrapolation of values and 
ranges; this might prove valuable for certain evaluative comparisons 
among instruments or data from the various instruments; and, c) The 
costs associated with drawing of multiple samples would be reduced 
and this could perhaps allow design of more representative and more 
efficient samples in the first place. 

Use of the same sample, or of overlapping samples, for the application 
of multiple instruments would be reasonably sound from the point of 
view of representativeness of the population, at least, as long as 
the reuse of the sample were in the same year as it was originally 
developed. During a single year the populations of most districts, 
schools and pupils would not change by an anount important enough 
to demand a new random sample draw; and if one r s sample were random 
the first time used, it would be random the second time, also. 

Furthermore, even if the populations did change within a year, updated 
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information with which to draw additional samples would usually not 
be available within the year in a form suitable to be used in a new 
draw. 

General Guideline for Sample Development 
The 1970 sampling requirements call for the generation of 
Elementary school samples, separately, which are representative on a 
State by State basis (or within a State). These samples are to be used 
for the administration of the PCI's within the Belmont States# Aggre- 
gation of these to a National level sample will involve additional 
sampling of Non- Belmont States. Some general guidelines for the 
research/sampling design development are presented below: 

1) The desired evaluations are to be Local and National so the 

development of interrelated samples and designs is necessary . 

The" should include: 

(a) For each of the cooperating States separately, a design 
to allow direct comparative evaluation of school and 
pupil characteristics through the various data instru- 
ments of the Evaluation Package. This cells for a 
discrete sample, for each State, representative of the 
State to a degree sufficient for generalization at the 
level of individual pupils and schools within the State. 

(b) A design for the same evaluation capability on the same 
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characteristics for the aggregate of the Belmont States. 

It is presumed that the sample for this aggregate is 
simply the combination of samples for the individual 
States. However, there may be good reasons for con- 
structing State sub-samples to represent this aggregation, 
(c) A sample which allows comparative evaluation of character- 
istics of pupils and schools for the Nation, to at 
least the same degree of precision obtained for the other 
samples. It is assumed appropriate to incorporate 
smaller portions of the other (Non- Belmont) States such 
that these can be appropriately weighted in relation to 
the 3elmont States to collectively represent the National 
population. 

2) These samples, should be linked through School Districts to 
limit the administrative loads of USOE, States and Districts. 

3) The research designs must concentrate on the Programs and 
their individual components as major U tors for the stratifica- 
tion or Identification of different groups of pupils and/or 
schools in order to assess the differential effectiveness of 
various Federal Programs and funded activities in the schools 
and Districts. These samples will be used to answer research 
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questions of Interest to BESE. The basic research questions 
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to be answered are Indicated by the evaluation system objec- 
tives! the system is to assess and compare the pupils and 
schools within and among States and between and among Program 
activity classes. This requires the data collection on a 
sample of the defined population of pupils and schools 
corresponding to the types in question and operating under 
the conditions of interest. Whether all such questions could 
be answered through applicat ion of a single design* is 
unanswerable at present. It ij certain that, given the 
universe of schools and the administration of the system 
instruments (to interrelated or discrete samples), the sample 
can be defined to meet specified reset, ch question requirements 
and can allow collection of adequate date to answer the ques- 
tions , 

Sample definition to collect data to answer such questions 
requires precise formulation of the questions to be answered, 
specification of the data types required to answer them and 
definition of the level of precision of description desired. 
These items are required because they determine what the basic 
unit sampled will be and how many units will be sampled. 

The level of precision of description desired determines 
the units to be sampled. For example,a recent study developing 



154 - 



a sample of elementary school students In fourth, fifth 
and sixth grades for the re-standardization of the reading 
subtests of commonly used Achievement Test Batteries used 
the school as the sampling unit. This was done in this 
instance because the unit of generalization of the data was to 
be children, Conversely in the 1968 and. 1969 Title I Surveys 
of Compensatory Education and for the 1970 CPIR and PCI 
applications interest was centered at the District level, 
since it is mostly at the Districts that the various 
differences in funding occur. The basis unit is the district 
for those designs. 

When the sampling unit has been determined, the next 
question involves the number of units to be sampled and the 
manner in which they are to be sampled. Stratified random 
sampling should be utilized, and tha size of the sample 
drawn should be determined by variables measured and the level 
of precision required. Allocation of total sample size to 
strata should be by the optimum allocation method. 

If Project Reference File is developed it will be an 
ideal sampling frame, since it will allow location of 
those schools, districts or States with programs of Interest 



to them. In addition it will contain information which allows 
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for meaningful stratification of the population. 

The Project Reference File is to contain the following 
data types for each school entry: 

1) State and County of location 

2) School mailing address 

3) School operating status 

4) School District name 

5) School District superintendent's name 

6) Name of person responsible for Federally funded 
programs 

7) Area code and telephone number of #5, above 

8) Name of school building 

9) Grade span of school 

10) School membership by grade 

11) Number of high school graduates for previous year 

12) Degree of urbanism of school attendance area (four 

categories have been suggested for use: rural, 

small city, suburban, large city). 

13) Lists of federally supported categorical aid programs 
in v;hcih the school participates. (Either for the 
total school or by grade). 

14) And such other information which the State may 
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desire to enter for schools. 

Among these data types, the following variables would 
be meaningful in stratification of the population for sampling: 

1) State and County of School Location 

2) District Name (and Size) 

3) Grade Span of School 

4) Enrollment (Total or by Grade) 

5) Size of Graduating Class Last Year 

6) Urbanism of School 

7) Program participation (by Program and Title.) 

Referring to the 1970 National and Sta e sample needs, 

and ignoring temporarily the National Sample requirement, 
sampling will occur within a State. The following factors 
are of major concern in such sampling for the indicated 
reasons : 

1) District Size (and location): Districts are fre- 

quently equivalent to counties and are always geo- 
graphically distributed. District size variations are 
very large in some Star.es and these variations 
are indicated to be related to variations in 
academic achievement and other variables of interest. 
If stratification by size is used appropriately, this 
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will group the more similar clusters (districts) into 
the same stratum and this will reduce the between 
cluster variance for the stratum, and thus Increase 
the accuracy of estimation of parameters. 

2) School Enrollment: If Schools within Districts can be 

stratified by enrollment /grade, this will further contri- 
bute to reduction of the within stratum variance for 
those variables correlated with school size. 

3) Urbanism: While highly correlated with Districts (due 

to locations), this variable is also somewhat correlated 
with socio-economic status, intelligence and academic 
excellence. This factor should at least be considered 
for use in stratifying the population, 

4) Program Participation: This variable is one of major 

concern to the total USOE paradigm. Stratification on 
this factor will work more effectively than any other to 
reduce the within- stratum variance of measures which are 
attributable in whole or in part, to differences in 
program participation. In other words, when variables 
are affected by Programs, stratifying on programs will 
yield the least error of estimation for those measures. 

If any one of these factors were to be discarded in favor of a 
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Smaller matrix of cells it should be the Urbanism factor since 
this is partially overlapping (due to locations) with District size, 
School enrollment would be next factor suggested for elimination from 
stratification use, since it also overlaps in potential variance 
reduction with District size. Thus, a highly reduced set of two 
stratification variables might consist of: 

1) Program Projects, and 

2) District Size 

State and National Research Designs 

Given the requirements for State and National Sample/Research 
Desings as outlined by JOES implementation needs, it is possible 
to specify the approaches which can be taken to define and determine 
the eventual designs which should and can be developed. It is not 
possible to provide final sampling plans for these samples since no 
Universe File or Program Reference File, data are vet available . 

The following set of guidelines for State and National sampling for 
U60E's current problem have been derived from the consideration above and 
the basic needs to provide estraates at the level of pupils within schools. 

1) Sampling should be by a three stage cluster sampling 
method to lower basic costs of the sample and data 
collection, and to provide adequate representation of each 
State and all of the cooperating States within che Nation, 
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2) The first stage clustering should be from a stratified frame 
based on the School Districts, stratified by size and by 
Program participation. (District sampling is Indicated by the 
results of the 1968 Title I Survey.) 

3) The second stage clustering should be of schools within the 
stratified District. Schools should also be stratified by 
program participation level, If possible, to provide data 
from schools with and without Programs. 

4) Third stage sampling should be of students within grades 
within schools selected from the primary units, the Districts. 
The selected grades should be such as to yield data comparable 
to that derived from the earlier Title I Surveys. Blementary 
Grades suggested are the 3rd, -4th, and 6th, (overlapping the 
Title 1 surveys' 2nd, 4th, and 6th, but avoiding the observed 
lack of data availability for the 2nd grades). 

5) Pupil sampling should be by ssmpllng within each class section 
of the desired grades (this avoids the Introduction o f the 
fourth-level sampling of classes and then students.) 

These guidelines summarise the approach which could be taken in 
this derivation of each of the Belmont State samples and also, with 
different levels of sampling in the non-Belmont States, for the National 
sample. Title general approach la for a three-level sub-sampling of 
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students, within schools, within Districts. It is also Intended to 
obtain a self-weighting, epsem, sample of pupils (epsem - equal pro 
bablllty of selection method.) 



i 
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REQUIREMENTS FOR FURTHER DEVELOPMENT FOR JCES IMPLEMENTATION 

This section presents a brief summary of the major develop- 
mental efforts remaining for the proposed implementation of the 
total JCES in relation to the Federal/S tate/Local Delivery 
System. There is no attempt herein to develop the specific 
approaches to the further developments. Some of these have been 
suggested in the previous sections of this report and others have 
not yet been attacked. The following paragraphs consist primarily 
of, a listing of the obvious and severe needs for this development 
and brief discussions of the needs. 

R eport Content and Format Specifications 

This requirement is a most critical one for the further 
development of the total JCES. As discussed under System Reporting 
Requirement?,, previously, this requirement is critical to several 
i>ther areas; to wit: 

1. Detailed reports specifications are necessary in order to 
be sure that the various JCES Instruments encompass and collect 
all of the data elements required to present the desired informa- 
ft ion to the managers and evaluators who are users of the outputs 
of the system. 

2. Detailed report specifications are needed for the final 
development of the Master Data Analysis Flan (see below, also), 

As indicated under Haster/Data Analysis Plcn Development, 
previously, these specifications should be the basis for the 
detailed analysis plan development since they comprise the basic 
informational output requirements that the system is expect'd to 
t rovide. Although MDAF development could proceed without detailed 
report specifications, (for example, as based on best 
,f guestimates M of any and all analytic processes which might be likely 
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to be required), the detailed report specifications development 
would greatly simplify and speed the MDAP and eventual data 
processing system developments. 

3* Report specifications appear to be especially critical to 
the definition and development of the State Educational Agency 
Management Evaluation Survey (SEA-MES) , With the development of 
this survey or set of instruments in a very preliminary stage, the 
early definition is needed of those reports essential to allow 
the management system to best understand its own operations and 
to evaluate different approaches which might be tried out in 
various agencies, The developing SEA-MES concept can be furthered 
greatly by an approach which defines the specific data needs of the 
survey(s) as early as possible. This would also contribute to the 
completeness of the MDAP development, since the MDAP should be as 
all-inclusive as possible to provide the optimum basis for develop- 
ment and lmpUmentation of the data analysis and data processing 
system necessary to support the over-all JOES efforts. 

A procedure was suggested previously in this report for 
development of detailed leport specifications, Including both 
formats and detailed information requirements for the various 
individual data sheets to be generated to support both management 
and evaluation functions of the JCES and USOE/State Belmont 
Organisation. It is recommended that the suggested procedure or 
something aimed at more efficient accomplishment of these same 
alms be instituted as soon as possible to further this critical 
developmental requirement. 

Required Development of JCES Instrumentation 

Several of the propored JCES instruments remain to be finalised 
at this time. Actually, the CP1R and the Local Evaluation Handbook (AIR 
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Guide) are the only newly developed instruments which are complete 
(excluding here the ELSEOIS and the Universe Pile). The PCI's for 1970 
have undergone extensive revision from the 1969 Compensatory Education 
Survey instruments and should he finished (final draft) on/about 16 
January 1970. Assuming that final review by the Belmont Group does 
not result in significant requirements for changesi this instrument 
should be ready for clearance, processing, and later din'emination 
to schools t.X approximately the desired time - April 1970. 

The Project Descriptor Instrument is etill in draft form, 
but is being further evaluated and is to be refined into a more 
streamlined instrument if this proves feasible. Contractual develop- 
ment of this instrument is planned for completion during January 1970, 
and aesuming USOE approvals, clearances, etc. should be ready for the 
planned implementation in the Fall of 1970. Whether further refinement 
of the end-product of the contractor's effort should be undertaken 
by the Joint Federal/state Taek Force on Evaluation itself, is still 
in question due to the rather unwieldy aspects of the latest draft 
instrument reviewed Continuing contractor efforts nay well alleviate 
these problems and result in an instrument practicable for installation 
as part of the JCSS. 

The Current Status Measures (CSM) instrument is in a some- 
what similar situation to the PD1, in that ie etill in a 
rather preliminary draft stage and there are some serious 
questione regarding ite appropriateness for use. The major 
problem concerns the content of this instrument which appears to 
contain some serious cultural biases within the item pools prepared 
for each of the two intended scales. With respect to the contents 
of both the Basio Verbal scale item pool and tta for 
the Occupaticnal Cognisance scale, one can seriously 
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question the degree to which they are biased toward concepts on which 
white, native, North Eastern or Western Americans would tend to score 
conslstenly higher than other ethnic or even geographic groups might. 
The degree of severity of any such biases cannot be easily 
estimated, but they would appear serious enough to suggest that the 
proposed 12 - Item scales to be built from these item pools would be 
prone to yield grave differences among large groups which might 
result from the biases rather than from real differences. 

The question of real or apparent differences in CSM scores among 
groups raises another 3erlous question about these scales: - they are 
very short, and thun may be expected to have low reliabilities. Witl) 
instruments of low reliability, dependable conclusions are difficult 
to come by, even for groups, and of course such instruments should 
never be the basis for conclusions about individuals. 

These questions about the constructs, content and utility of 
the CSM scales as currently developed should be considered with re- 
spect to possible revisions. Whether these scales could be used, 
as is, in spite of the current inherent difficulties to gather any 
useful Interpretable data is a moot question. 

The last instrument, the Program Reference File, is intended 
to be an expansion of the content of the Universe File collection 
of data on school activities lr Federally funded programs. It 
was originally Intended to substitute for the Universe File 
questionnaire. This was disallowed within USOE, and the next concept 
was to apply the additional items to the Belmont State schools 
population. Whether this is currently to be done or not is still 
undetermined. The purpose of such additional data collection was 
to provide program participation data, by schools and by grade level, 
together with data on the urbanists of the schools, for use in 
stratification of districts and schools on these factors. 
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This approach would substitute somewhat more meaningful factors 
for the currently-used enrollment size of school as a major 
stratifying factor. 

Refinement of Sampling/Research Plans 

The research/sampling plan developments for the Implementation 
of all survey Instruments require updating or completion at this 
time, since even those for the PCI's (for April 1970) and the CPIR 
(Fall 1970) require expansion under the current plana. Furthermore, 
the development of representative State samples, whether within the 
Belmont States or Nationwide, might necessitate some slightly 
different considerations as the bases for sample development, 

In any case, the existing samples do require augmentation for the 
purposes of Intended applications In the coming months. 

Master Data Analysis Plaft Development 

As Indicated earlier In the report, the develf/ment of the 
analysis plans has already been undertaken for some of the 
Individual Instruments - notably, the 19^9 Survey, which Is closely 
similar to the 1970 PCI. However, even with precise Individual 
Instrument analysis plans, the necessity would remain for the 
Interrelation of th?s«, plans, and for the development of a master 
plan for Integrated analysis and reports generation. This Is a 
critical remaining requirement, especially In view of the desire 
to Implement a large portion of the total system, Including the 
data processing capabilities within the next year. 

Data Processing System Specification, Development ard Implementation 
This aspect of system Implementation was briefly discussed 
previously, also, and It need only be reiterated here that, regard- 
less of the eventual extent of data processing capability which is 
settled on to support the JOES, no extensive planning or design work 
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has yet been undertaken* With the widely recognized tendency for data 
processing system developments to require two to three times as long 
as predlcted f this item is one which should be accelerated. 

Considerations Relating to the Management Syste n ; Including SEA-MES 
Several points related to the developing management chain 
within the Delivery System and the evaluation of the processes within 
the chain might be pointed out for additional attention. These 
Include at least the following: 

1. The SEA-MES development was mentioned above under the 
category of repot ts specification. Obviously, that is not the only 
aspect of this survey development which requires attention. Since 
this survey device is in only the embryonic stage, as yet, a full- 
scale effort is required for development and implementation of the 
Survey. A major consideration in this development might well be 
the specification of the reports typifying the processes and activi- 
ties on-going in State Agencies and the relation of these to the over-all Delivery 
System management chain. Ihis approach would also clarify the inter- 
faces between the Funding Subsystem and the Substantive Program 
Subsystem of the total Delivery System, Such clarification would 
as 1st the JOES Task force on Evaluation and the other working groups 
of the Belmont Organization in further defining the requirements of 
the SEA-MES. It appears that the SEA-MES might even become 
Involved in assessment of management activities at the district 
levels, as well as at the State and Federal levels. This statement 
is based on the large quantity of District to State reports and 
lntercommunlcat ions which undoubtedly affect, even dictate, some 
SEA activities. Over-all evaluation of Program Administration 
and Grants Management at the State level may well require some 
limited evaluation of these actlvltes at the local levels as well. 
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2. A major purpose of the Belmont Working Organisation in the joint 
venture related to establishment of the JCES has been the increased 
emphasis the JCSS might stimulate at the State and local levels on 
the development of staff members in areas of management and evaluation 
technologies. ThiB purpose has been undertaken on a limited basis 
through the CSSO'e and other State representatives! and interest appears 
to have been generated. Capitalizing on this growing interest and ths 
Belmont Organization’s Fedsral/Stats level could bring about both the 
planning and implementation of staff development practices within State 
and local levels in the near future. In fact, an Initial step hae been 
taken in this direction with the recent presentation of three one-week 
institutes on evaluation tools and techniques. These institutes, 
designed by the Belmont Oroup Committee on Evaluation Training and 
Local .Evaluation, were preeented in August, September and Ootober of 
196$r> It seems that the rucceseful extension of these concepts depends 
upon the development of a series of models for the following, which 
could be applied by the State and Local Agencies for thie purpose i 
^ a. Management and/or Evaluation - Staff Envelopment Goals 

b. Management Plane - directed at State and local level 
activities related to the management and administration 
of the Federal and State funding programs and the maxi- 
mization of funds utilization to increeee the payoffe 
for the States and local etudent populations. 

c. Management/Evaluation Training Software - materials 
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similar to the Local Evaluation Handbook (A1P Ouide) 
which would allow both individual and group action to 
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improve capabilities relevant to grants management and/or 
evaluation of both funding programs and substantive projects 
in schools. 

d ■> The set of items above could lead to the development of 
a series of USOfi/CSSO sponsored Management/Evaluation 
Training Institutes or Workshops to be delivered at various 
State and Local Agencies as requested. These could be 
direoted toward speolfic performance goals or toward 
general i sable skills and knowledges in these areas. 

3. In conjunction with the last item, above, it is suggested that 
the Joint Pederal/state Task Force examine the possibilities for insti- 
tuting the requirement that the combination of the Project Descriptor 
Instrument (or an abbreviated version thereof) and the Local Evaluation 
Hnadbook (AIR Ouide) be used in preparation of the Local to State 
narrative evaluations of Programs and Projects instituted under State 
Funding management. Similarly, the USOS Bureaus could investigate the 
potential for ready acceptance of some similar documentation to guide 
the preparation of the Local reports to USOE related to the evaluation 
of the Projeots developed under the Direct Oranta Programs. The utilisa- 
tion of these two instruments in evaluative reporting to upper management 
would undoubtedly increase the standardisation of reporting and thereby 
( increase the effective capability for comparative evaluation of Program 

, and Project activities at both the State and Federal levels. 



Mi ni 




\ 



